Charts and charts (20090803)

Escharts support. Finished the payment handling. Working on the prepayment code - which involves expanding the web page to accommodate a payment provider selection, and underlying code to do what's needed (as per register.aspx) for each payment provider. Currently the page
only supports a single provider - so I need to split out the invoice and order generation code from the page load code, and stick in a couple of post back buttons (one for each provider) and do provider specific stuff in the postback handling code for each button.

Creating a win 2003 server VM, running 2 websites, one for charts.bubble.com and another for escharts.bubble.com.

Callbacks and state variables (20090731)

Matching Worldpay callback code - making sure to exclude/extract code for other operations - cancellations etc.

Slight hiccup in the PayPal/MIS implementation is that as I uncover each requirement from the existing code, and implement it for PayPal/MIS, I discover a bunch more requirements. All of this makes trying to determine how far there is left to go rather difficult.

Worldpay also supports accepting any number of passthrough variables (and values) which the shop code uses to sling a bunch of state data and flags around. PayPal doesn't support this - so all this information will have to be saved into the DB before any call to PayPal. This I will make as generic as possible.

Extra Subscription info (20090729)

Implemented the PMISProviderData table and class (using nORM) The SetExpressCheckout now saves the provider data, and GetExpressCheckout loads it again.

Also saves the payment id (returned from PP) into the invoice table.

Trying to work out the kinks in specifying a trial period for the PP recurring payment profile - the combination of period (day), frequency (14) and total cycles (1) causes a PP error, about the total cycles having to be bigger than 0 (which it is.) Solved for the meantime by moving the start date to 14 days from now, and removing the trial period.

Tracking down how the mechanics of user service subscriptions work - and what exactly the lifetime is of data in the tblUserServicesSubs table.

Added a separate subscription data table, which gets populated with userid, serviceid and provider id (which in the case of PayPal) is linked to the recurring payment profile.

No "Custom" for recurring payments (20090728)

Battling with merging various repositories from Hg - which I've decided I need to do to keep the differences from mounting too much.

Integrating nORM from MUDI2008-parallel, into MUDI2008-mis.

Successfully creates invoices using PMIS for PayPal purchases. Creates an invoice with zero amount for subscription only order.

Having a problem tracking down why PayPal is no longer returning the "Custom" field (which contains our invoice id, cart id etc) and is used by the callback to know which order etc is being dealt with. This was happening yesterday, and isn't happening now - and googling for "getExpressCheckout" and "Custom" doesn't produce anything useful.

Turns out that PP doesn't support the "Custom" field for recurring payment only transactions. Going to have to look at using the PP token (returned from the first PP call) to index a local store of all the information we currently store in the "Custom" field.

Looked at using "SetCustomerBillingAgreement" instead of "SetExpressCheckout" for a recurring payment only transaction, but ignoring it as: 1) The documentation says don't use if for later versions of the API, and 2) It doesn't support the use of "Custom" for recurring payment only transactions.

Recurring payments

MIS/PP - discovered that because the paypal website doesn't display payment details to the user, and actually displays a message saying that they will confirm their payment details on the following screen, I will have to jiggle the order of the code a little. Currently, we display the summary to the user (including amounts etc) and the Next button takes them off to PP or WP. We will now have to shift that for PP, send them off to PP, and on the callback return, display that page, and the Next button will finish the PP transaction (the code that is currently in paypal/callback.asp.vb)

Working through code to determine why the instant payment for the service subscription is zeroed, and how we can prevent it for PP users. It only seems to happen if a service subscription is the only invoicable item in the order. Sorted - seems to be something I did - or was looking at the wrong data.

Adding basic recurring payment setup. Need to make changes to setExpressCheckout to set up recurring payments in the first place (not quite sure why - but PP demands it)

Looking for validation (20090302)

Uploading atlas validation data. Then rerunning the atlas validation data generation, as the original only had around 2000 cities in it. And once I discovered that I hadn't been able to update the original atlas data correctly, I managed to get that done, and re-run the validation data generation. Definitely the last time. Perhaps.

Working out the best way to combine a set of date spans, given a set L of date spans: L[(s1,e1), (s2,e2) ... (sn,en)], I need to cycle through the set and merge them down into the smallest set of spans.

I also need to keep track of how many times each portion of a date span occurred within the original set. This is so that the weighting of each date span can be calculated. The simplest way may be to use a sparse array of all possible dates, the largest size is pretty much 80*365 (80 being the maximum age we look at.) Then increment all the dates that are covered by a span.

I need to work on some way to speed up the condition matching. Those that involved a progressed midheaven and ascendant are quite slow/inefficient, mostly because it needs to step through a large number of dates (365*80 on average) checking for a match on each date.

Ended up running the atlas validation just one more time after some tweaking to make sure that we extract the correct county data every time (it seems to be handled differently for different countries)

The Beginnings of Fertility (20090226)

Lots on new Ubuntu VMs. Working the new biscuit OF VM, trying to get the user-context switch working so the instance can start up as root and then switch to ofop, after opening the listener on the right port (443)

Worked up a script to stick into /etc/init.d to start the instance. Basically took the init.d script for ntp, and fiddled with it from there.

Found out the easiest way to get the service started at the correct run-level (on Ubuntu it's update-rc.d)

Started working through a check of all the fertility checks to make sure all are valid (it's been quite a while since I last looked at this code.) There was quite a bit of confusion initially as to what we are actually checking, and I need to make sure that we ended up implementing the correct ones. Once we're happy that we are doing the right thing (as far as all the conditions are concerned) we can implement an interface for VB6 to use.

VMware (20090225)

Attempting to set up some automated tests against MC, using wget, which is having problems connecting to OF. Wget fails with a 'Read error (Cannot allocate memory) in headers', and simply retries. I've attempted this with a local tunnel, to bypass the SSL side of things
in case that was causing the problem, but with no success. I've ended up using lynx, which is a bit slower (not sure why) but does the trick.

A bit of investigation into backup MX servers, using Postfix, and using umask to force the permissions of newly created files. Testing umask and g+s changes on uk server, which didn't work very well. Seems there are documented issues with scp honoring the umask that is
set in .tcshrc or .bashrc. As it turned out, we can use find, with -not and -perm to find any offending files, and fix the permissions via a cronjob.

Fixing a sizing issue with images generated by Ballista. Added height and width parameters for each of the 4 images generated by retrieveNatal (just to break the API - insert maniacal laughter here.)

Set up OF on a new Ubuntu VM on holly, called biscuit which will run the old admin instance (once I get hold of the OFORs and config files.)

Network problems (20090224)

PHP extension work. Got a basic extension working locally - now I just need to get it talking to Ballista.

Rolling out a new Ubuntu VM on Holly, for the admin OF instance. Had a lot of problems with network connections (ssh + scp) getting dropped. Still having these network problems, so changed the IP address on the VM, just in case it wasn't spare, but the problems persist.

Extending PHP (20090223)

Fixed the odd build errors on Linux that appeared after the modifications to the source required to get the Win32 build back working.

Spending a bit of time thinking about how best to implement to local ballista service/cronjob.

The immediate options are a single app that runs through each user and updates the MySQL database with new transits etc, or a controller application that scans through the users, and passes off the calculating and updating to a separate application/library. This second option will mean that it should be easier to implement a single user update for the PHP code to use for new users. It means we can use a PHP command line script as the outer scanning application.

Attempting to upgrade/configure holly - which resulted in a reboot, without sshd starting - so attempting to get hold of the drac/lights out details to connect to the console. Turns out it took ages to start, and I've managed to upgrade vmware (and configure it successfully.)

Work on a PHP extension to interface to Ballista. The PHP extension API and build environment don't look too complicated.

Stupid Software Companies (20090217)

Adding the new API to ballista to expose the planet positions etc.

Extending the new web service to return natal aspects and house positions.

Spending a really stupid amount of time dealing with inconsistencies in the piece of rubbish Microsoft call a development environment. Whilst debugging the web service, sometimes the debugger will let me step into an underlying DLL, the next time it won't, with nothing changing (apart from a restart of the same debugging session.) Sometimes rebuilding the underlying DLL helps, more than often - not.

I'm sure someone must have done some research on how much time is spent dealing with the development environment Microsoft provide, and how much time is spent on actually debugging.

Had more fun with Microsoft, when, after copying new DLL to the test VM, got the most informative message in the world "The application failed to initialise (0xc0000022). Click OK to terminate the application." Absolutely wonderful. No enlightening message as to why it terminated. Some googling turned up that 0xc0000022 meant access denied. One would think a message "Access denied trying to open 'X'" would be useful. I must be too stupid to work with
Microsoft products. In the end, using Filemon, I finally worked out which file had a problem, and which user was trying to open that file.

Deployed the new web service method to the test VM, and sent out a mail about same. Finished the evening by throwing more ebola infected darts into a MS Windows XP OEM package.

More activation and Ballista (20090216)

Ironing out some issues with activation, and existing users that haven't been activated. The handling of activated/not-activated is getting over-complicated (and probably not very efficient)

The 'activation has been sent' message also needs some prettying up. I've used the standard header and footer includes to match it to the normal static pages.

Started working on a single Ballista webservice method that will return all natal images, planetary positions, and relevant transits for the current 3 day period for Jack to use, instead of him having to deal with 3 different web service methods. Involves adding some new code to ballista2 to access the planet positions (along with ascendant and midheaven.)

Activation (20090213)

Thoughts on the activation email. The simplest solution is to not allow logging in until the account has been activated. The problem with this approach is that we automatically log in the user during the registration process, to allow him/her access to the birth details forms.

We can allow access to an un-activated account, until an activation email has been sent out. From that point on, the account is locked out until the activation URL has been clicked on.

For a new registration, we allow the logon (as it happens away from the normal log on code). Then in prefilter, we need to check if we are activated, and if we aren't, depending on request URI, redirect to a message about account activation.

The basic email activation is in place, with handling of the odd situations where the account isn't activated, but we need to be logged on (setting birth details.)

Investigating how to send HTML mail from php.

The handling of a user clicking on the activate link in the email is done. Now, after sorting out some fundamental differences between isset and is_null, the activation code seems to work just fine.

Between the logging on process and the activation process, the prefilter code has got quite cumbersome. I'll spend some time this weekend having a look at a better way of streamlining the code somewhat.

Another blog

I've set up a new blog containing more generic posts, less detailed discussions of actual work, and more detailed discussions on how to get things done in a more general fashion.

http://weaselburrow.blogspot.com/

App Engine and Captchas (20090212)

Investigating the terms and conditions of GAE for the use of preview2. Especially the scalability (loads of emails once we pay, we get about 2000 a days for free.) There is a local application development environment, which can be used for local development, before deployment. The only downside (guess that depends on how you look at it) is that only Python is supported at the moment.

Added configurable different colours for each type of aspect in the natal wheel. Currently they are all different colours, but we can leave most of them one colour, and highlight any particularly
important ones.

Deployed the new getinfo page as the 'Welcome' tab in the homepage, which displays the user's full name, their place of birth, and both birth date/times (local time and adjusted GMT.)

Investigating PHP captcha implementations. Found one that seemed to fit our purpose, but had some quite severe problems getting it to work within our framework, and I've ended up yoinking the code (it was GPL) and integrating it into our framework.

UPDATE: After some experimentation and fiddling with configuration, I've managed to get my local PHP working with msmtp to send mails out (removes the need for sendmail or similar on local machine)

Back porting again (20090211)

Finished the Ballista port to linux. Up now are two things: A cronjob, running some client that will update everyone's upcoming transits. It can also handle sending out mail etc if necessary.

The second is an interface from php, so that the new user code can use the local ballista, instead of using a web service to generate the natal images, planet locations etc.

Had to redo the linux port after I got mixed up with using the wrong set of source code - although it went much faster than the first time as all I had to was do an 'ediff-files' on any source that broke the compile.

Local transits (20090210)

Made and deployed some fixes for tracker.

Working on porting the ballista code to linux - with a view to running the transit generation code locally to either preview or dyna.

To get the cross-platform functionality I need, I've used the OF system library (ofsys). Having great fun with stupid little issues, like trying to link against certain libraries instead of statically
including the source (sqlite3 is a good example.)

The introduction of ofsys has made a lot things a lot easier - there has been a great deal of testing, especially the cross-platform side on ofsys. There are still some issues as to things like where the configuration and data files should live

Indexing (20090208)

Working on updating the edit/new user pages to use the same background, style etc as the normal sign pages - so as to better fit in.

I've created a header.php and a footer.php, which drag in the background image and logo, and can be used to set up any global styles.

Investigated the best way of creating indexes under MSSQL for use by the soundex functionality.

Added indexes (fulltext and normal) to local Atlas VM, which greatly speeds up the town/country search. Having great fun trying to create a full text catalog for MSSQL on the Atlas VM. SQL Server Management Studio has got to be one of the worst designed user interfaces on the face of the planet.

Rolled out the new code to preview2 and dyna.

Magick and issues (20090206)

Finished rollout of the issue tracking system. Doesn't have a DNS entry as of yet.

Changed the image directory handling to use split the user identities into thousands, so each image subdirectory will only contain images (and any other user attachments) for a 1000 users.

Worked out how to use transparency with the Magick++ interface, and generated some sample transparencies for a decision. I've also changed the API of draw_house_segment in the image backend to accept a transparency fraction (0.0 - 1.0) and stuck it into the configuration (under house_transparency)

Deployed all the latest source to preview2, and removed all the registered test users.

Fiddling (20090204)

Re-arranging the birth details form, sticking the place of birth fields (city and country) at the end of the form.

The welcome banner is now resized, placed where the logon box would normally appear, and reformatted to better fit.

Working on the generation of the natal_wheel image. Not much to see yet.

Also set up a basic issue tracking for keeping track of what is still outstanding, and to keep notes on each of the outstanding bits and pieces.

Bit of a slow day - results wise, lots of fiddling to get not much done - resulting in a frustrating time.

Ordering possibles (20090203)

Added a cookie signifying logon failure back to the static main page, and javascript there displays a 'Sign-in failed' message if it is present.

Working on dealing with cities like 'Kowloon' - which don't have a direct match as 'Kowloon City' is actual city name. It appears in the soundex search - but that isn't sorted either.

I'm adding a 'reorg_possibles' function which will sort the combined direct and soundex matches, this is based on :

Given CITY and COUNTRY - we get a list of possibles+soundexes back from the the atlas web-service. These are a list of PCITY and PCOUNTRY

If PCOUNTRY is a priority country (UK, England....):
{
  If PCITY matches CITY - actually Left$(PCITY, LEN(CITY)) = CITY
     Add PCITY/PCOUNTRY to array_match
  ELSE
     Add PCITY/PCOUNTRY to array_country
}
ELSE // The PCOUNTRY isn't a priority
{
   If PCITY matches CITY (As above)
       Add PCITY/PCOUNTRY to array_lower_match
   ELSE
       Add PCITY/PCOUNTRY to array_lower_country
}

Then combine the arrays into a list in the following order:

array_match, array_lower_match, array_country, array_lower_country

So for London, you'll get all the London's in the UK, then Londons in the rest of the world, then near London matches in the UK, then near London matches in the rest of the world.

For Kowloon, you'll get Kowloon City, then near matches in the UK, followed by near matches in the rest of the world.

I've redirected a successful registration through the external site, using the redirect.php to load the parent frame.

The new_user2.php page currently uses two forms, one after the other. The first contains the birth date and time fields, and the city and country of birth fields. The second contains the listbox with the possible matches. Both forms submit to the same page, but were kept separate to make sure the php code was picking up the correct submit button. In order to re-arrange the fields (sticking the birth city and country fields under everything else,) I'll have to merge them in
to a single form, and work on making sure that PHP can pick up the correct submit button.

Polishing (20090202)

Polish time for the deployment on preview2. Created new 'sign in' and 'register' images. Cleaning up the form code in main.htm template, and the javascript in general.js.

The successful registration (and birth validation etc) results in aredirect to index.php, which needs to change to get the parent windowto reload itself.

Implementing a better session cookie - one that contains the user id, and one that can supersede the PHP session cookies, which is expiringlong before it should. I've done this, using mcrypt, to encrypt the user id into the static site's session cookie (set by the dyna php site.) If the PHPSESSION cookie has expired, or the session itself, which seems to be what is actually happening, and the static site's session cookie is available, it is decrypted to extract the user's id, and the PHP session is recreated from that.

Experimenting to find out how: divs loading/running there own javascript code. Despite there being a bunch of solutions found via Google, most of them don't work. Googling for information on HTML, JavaScript or anything related is guaranteed to come up with loads of opinions, solutions or vague tidbits generated by complete and utter idiots. Forum responses along the lines of 'I haven't read your code properly, or tried this myself, but maybe this will work.....'

For the login/register functionality, I need to get the PHP code to somehow redirect or refresh the parent window, which should be doable via javascript. Done, done and done, I've created a redirect.php file which will use javascript to redirect the parent window - only doable as we're sitting inside a single host name (thanks to reverse proxy)

For tomorrow: The natal wheel generation is using the wrong config file, it needs to be set to use 'transit'. The automatic logon when registering a new user doesn't use the configured cookie expiration time, and a successful logon doesn't refresh the main page as it should.

Sine, Cosine and Sessions (20090130)

Things for today: Complete the deployment/integration with dyna.cainer.com, finally work out the placement algorthm for centering text around the radius of a circle (needed for the natal wheel,) storing sessions inside a/the database and work on the cron-job based transit retrieval for a three day forecast (using the separate retrieveTransits2 web service API call.)

Started off investigating PHP sessions, extending their life, and how we can go about storing them in a database, as they default to being stored in files in the /tmp directory - which won't work very well if we're firing off requests to different servers.

Discovered we can set session.gc_maxlifetime in php.ini, which defaults to 1440 seconds, which isn't really good as the underlying sessions can be destroyed before the cookie expires, so the browser will think it's logged in, and php won't. So we'll have to change that to something more reasonable. The next step is to save the session information to a database to handle load balancing. We already have the basic structure in place to handle different database
connections (restricting all requests with cookie x=y to database z) which should give any database replication a chance to happen.

Saving to the current MySQL database is done, after some fiddling around because I keep forgetting the PHP functions don't see global variables, unless you tell them the variable is global. There must be some way around this which needs investigating.

Some thoughts about the transit retrieval, which we'd like to happen on a daily basis. At present, this is done via calls to a web service. Also presently, this is pretty much stateless, in that the
call to get today's transits has to supply date and place of birth, with nothing stored on the side of the web service. If we're going to be handling 1000s of users (100 000 or 200 000) we will probably need a local shared library implementation of ballista, with a PHP api. In fact, a locally running process could independently update the transits within the database without any involvement from PHP.

Sorted out the planet and sign character placement for the natal wheel, turned out to be a bit simpler than I thought it would be, once I sketched it all out on a piece of paper.

Deployment (20090129)

Today I'm going to deploy what I have running locally on preview4.scrawnybat.com, onto preview2.cainer.com, and update the various scripts etc via Website Control.

Encountered several minor bugs: Permanently disabled email address field (should only be disabled when editing details) and a forced login halfway through the new user registration process.

Installed SSH server on ballista-ws VM to make my life a bit easier. Need to sort out the various ballista configurations lying around, to make sure that ballista-ws is using one that returns at least some transits. I've merged the configurations together, so that default.conf uses the future transit specifications developed by Steve. These are now deployed on the ballista-ws VM.

Deployed the new PHP backend onto dyna.cainer.com, need a couple of tweaks on the main.htm javascript to make sure it's using .php on the end of all the URLs. There are really any changes that need to happen on the FreeBSD frontend (it's still sending to dyna.cainer.com - and we just redirect vhost requests for that host to /cetla instead of /dyna.)

Transits and the homepage (20090128)

Making the decision on how to handle transits - it seems the best option is to fetch transits on a daily basis, using a cronjob, using the retrieveTransits2 API, which returns the most important transits for the current 3 days (centered around today.) The only issue with this is that it currently uses the local time (and zone) to determine when days start and end. Kinda implies that we should be storing the user's timezone when they log in/register. For starters we can use
GMT.

Created the Transit mini-ORM class.

Extracting the transits from retrieveData2 is done. This isn't the nice clean and localised (from a time point of view) set of transits as above - but it'll do for the moment whilst we get the transit
display up and running.

Basic display of overlapping transits (starting before now, and ending after now) done. Doesn't look very pretty.

I've got a homepage up and running, display 3 tabs (Welcome, Images, Transits), with a little Ajax and other javascript to display the relevant content.

Webservices and local image files (20090127)

Fixing the 'edit user' code, dealing with unchanged passwords.

The new UserImage mini-ORM class is done, suspiciously easy to build on top of the mini-ORM. We're retrieving the natal images (natal wheel, aspect table etc) successfully, and the web service supplies them in base64 encoded format. We can either store them in the database (as
either base64 or binary format) or write them to files somewhere, which is the method I prefer, if done correctly, then delivering them would effectively be Apache delivering static image files, with no PHP participation required.

Got the basic writing to a file sorted out. It does mean that there is a directory under the server which is writable by the 'www-data' group (or whichever group Apache is running as.) Each image is saved with the user id as part of the filename in a 'data' directory, and is accessible via Apache as a static file. There is a definite improvement in the delivery of images like the natal_wheel and aspect table.

Next is a list of upcoming transits. I need to merge the various configs we've set up for astro.conf, which has the best image engine config, and transit.conf, which was generated from a set of configuration items setup by Steve. Using that configuration set will give us a nice list of upcoming transits of some importance.

Mini-ORM (20090126)

Implemented a very basic ORM (it's missing the ability to determine the database structure at run-time - I fel't that would be too much of an overhead at this point - especially if I can't find a simple way of caching that information, it would be done once for every request.)

Implementing the staged 'add new user' or 'edit user details' in PHP, from gathering firstname, surname etc to town of birth with an Atlas validation to retrieving transits and natal images. By lunchtime I've got as far as knocking out the web service call to DBPlaceValidation.

Added automatic accessors (for reading and writing properties of records represented by the mini-ORM.) This means you can do things like $user->surname="Smith", instead of user->set("surname", "Smith").

The registration process, with the database atlas validation is all done. Tomorrow is the natal image and transit retrieve web service integration, and implementation of the user_images model (using the mini-ORM as with 'user')

20090125 (Actually Friday and the weekend)

After the decision to push ahead with a PHP implementation, I spent some time going through the available ORM implementations for PHP, with a view of using one from the beginning. After some thought here, I've decided that for the initial implementation, our database access is simple enough - and I don't really want the overhead of an ORM.

Next came the learning curve of understanding the basics of PHP classes, so we can have some sort of structure, especially within the data abstraction.

I've also switched to a MySQL database (we started with a SQLite3 database which is simple and quick to get going) as I reckoned I should be dealing with MySQL issues from the start. I've created the table definitions in sequential .sql files so they can be 'migrated' on the rollout server. As an aside, the two ORMs I investigated the most didn't seem to have a simple and easy to use database migration system (unlike Rails - which I have a feeling I will miss.)

I've now got basic user creation, and webservice client code working, the next step is to integrate them together, along with the existing logon functionality.

There continues to be a little re-writing of earlier PHP code as I discover better or easier-to-maintain ways of doing things, a really simple example is storing configuration items in a global array ($CONFIG) instead of a bunch of variables, although I will look for a PHP YAML implementation for this.

I've also set up a local instance of MediaWiki to stick all these gems into for future reference.