It started as a proof of concept prototype in the fall of 2010. The idea came from a meeting where we were discussing the porting of our storefront codebase from classic ASP to PHP. One of the discussion points was how to avoid simply porting the same logic from one scripting language to another, but rather finding ways to move some of that logic to other more suitable platforms, including service oriented solutions. The little program was written as a self-hosted WCF service written in VB.NET running on my Windows XP box as a console application. It implemented a RESTful API service that returned the number of products in a customer’s shopping cart. Really simple and modest in scope, the thing worked like a charm. I made a presentation about it during one of our Wayfair Engineering Lunch and Learn sessions about a week later. So far, so good.
Nothing happened to it for about a year, until one day in early November of 2011. The port to PHP had been successful and we were gradually moving traffic from our ASP scripted web pages to the new PHP platform. The holiday season was approaching and the PHP platform was taking more and more traffic every day. The web platform started showing lots of faults and things were getting worse with the increase in traffic. Turns out that the FreeTDS driver we use in PHP scripts to access our MSSQL databases had trouble making connections under high traffic conditions. The efforts to resolve the issues with the driver produced no immediate results, and we started to brainstorm for alternative ways to solve the problem. Wayfair Engineering had to come up with something soon. The biggest chunk of the DB queries affected were identified, and it looked like our inventory lookup query was the easiest one to divert to an alternative solution. Without inventory lookups, the FreeTDS driver seemed capable of handling the expected traffic.
A few ideas came up. One of them was to take that year-old WCF prototype and turn it into a proxy service for MSSQL inventory lookups. So I started adding the necessary code to implement the inventory query as a pass-through operation. So, essentially, the PHP client script would make a REST API call to the new service over HTTP passing the query as a parameter and get the query execution result in the response. I added code to implement the database connection logic from our standard method and tested that. I also needed to provide a good format for the response and wrote a function that formats a dataset object directly into a JSON object to facilitate the use of the response in PHP. Once all this was working I had to test it under heavy traffic conditions, so our DBAs provided captured query traffic from the production databases to use as realistic load. I wrote a simple client to call the service with the captured queries and monitored the performance, memory and CPU. At first it looked promising, but eventually the memory use rose to high levels, so I checked the code and found ways to eliminate potential memory leaks. After that, the service was ready for a trial in the production environment, so the PHP team built a client function and prepared to send the traffic of one of the web servers to the new service. The service performed reasonably well and more and more servers were added to the trial until all the inventory calls were using the service. The response time wasn’t as good as the one from the FreeTDS driver’s call, but it was close enough and it didn’t cause connection faults. The new service was named “http query”. Once the service was running in its own application server, it took care of our high traffic worries and sailed through “Black Friday”, “Cyber Monday” and the rest of the holiday season workloads without any major issues.
This experience made Wayfair Engineering aware of alternative ways to solve scalability problems. Since then, a few new service oriented solutions have spawned from the little program that could. One solution uses the same concept to process order data to identify fraudulent trends. Another effort by the Recommendations team built upon the same foundations to implement a more efficient way to retrieve inventory status for multiple products in a single request.
The Wayfair Engineering Express has left the station, and I’m so glad I got a great window seat. Long live the little program that could!