Heavy lifting Aussie style with aMiSTACX
When Joe Ritson approached aMiSTACX a month ago, he had already been working with aMiSTACX stacks on AWS for over a year. Like many of our customers that run Magento or WordPress, they have been victim of poorly designed modules, had nightmare experiences with offshore development agencies or developers, and were literally at the end of their rope.
aMiSTACX was the last bastion of hope before making the costly mistake of moving to a highly expensive managed solution like Shopify or Magento Commerce. Even though we offer development services to our customers, many don’t realize we are experts with Magento, WordPress, and network infrastructure, and can really make the final difference between a DIY or a managed solution.
Team Mission Impossible
aMiSTACX likes to call our DevOps and Development team Team Mission Impossible, as many of our projects are projects taken over where many development teams have previously failed, and the aMiSTACX Team must succeed.
Part of our success is we only take projects where we are almost 100% in the position to drive, we are NOT on a strict budget, and we MUST use all aMiSTACX products. All of aMiSTACX products are designed to work together in order to deliver simplistic management, and fast and stable performance. [It’s almost impossible to give accurate quotes on how much a project will cost, because each project is different. Each project has it’s own faults and gremlins, and each business has a set of must-haves.]
What we do know, is we don’t bill you to breath, we are extremely fast, you will be super happy when we are finished, and the bill will be a lot lower than you thought. [In comparison to other development teams.] Your sales will probably increase enough to cover the expense.
Six years ago, Australia’s first 100% veteran owned and operated fitness equipment company, was forged by Andy Taylor from a career of active service with the Australian Defense Force.
Aussie Strength was created with the goal of providing highly effective functional training tools to the Australian Defense Force, gym owners, and to fitness consumers world-wide. Read more…
What Aussie Strength needed now was to call in aMiSTACX’s Mission Impossible Team to pull off an entire refit of aussiestrength.com.au in order to fix bugs, improve performance, and to improve sales.
The Aussiestrength.com.au Project
Joe was technically savvy, and had his wish list. He wanted to at least go from Magento 2.3.0 to Magento 2.3.5p2, and he wanted the high-performance of our S3 Titanium G5 stack for Magento. Additionally, he wanted to go with Algolia Search, as I had previously sold him on that search engine after he took a peek at our customer’s Magento million sku catalog site, supplyapp.com. He also had a bug list of about 10-15 items that were causing countless missed sales opportunities, and degraded site performance.
A Plan of Action from Experience
Once we were given the green light, the aMiSTACX Team moved quickly. We first cloned an image of production [no one works on live e-commerce servers!], selected our Magento 2.3.5p2 G5 S3 Titanium LAP stack, and then migrated the magento data to its new home.
He was previously using a G4 LAMP setup, but for high-availability and for ultimate flexibility and performance, AS needed an AWS driven RDS database. Our new G5s are awesome! Blueprinted from more than three years of high-performance experience with Magento, the one selected was built on an Ubuntu 20.4 LTS server, Apache 2.4, PHP 7.4, and made use of an RDS MySQL 8.
Technical Notes: Although we have both Apache and NGINX HTTP server stacks, we normally go with Apache because it is so easy to use, and requires almost no maintenance. Additionally, from our tests in the past, NGINX vs. Apache, we were benchmarking only about 200MS difference in performance. We also favor the AWS T3 EC2 class, because of cost and performance. Since we have several layers of cache, even 2K unique users per day is keeping EC2 utilization between 5% ~ 20%. The T3 series also offers an [optional] unlimited bursts if you require or are concerned of potential lags under extreme loads. aMiSTACX wants to save money, not waste your money.
The Development Environment
Very much boilerplate for all our projects: Windows Utility Server for RDP with IDE phpStorm, GIT, Redmine Project Management server, and Area 51. This allows complete control of security of the entire development process. The workflow is very simple, we connect to our cloned server that was renamed aws.aussiestrength.com.au, we create a connection via Storm, and then when the project is completed we leverage a remote GIT repo to act as the go-between of dev and prod.
Implementation of the Plan
Once the development environment was all set up, our first order of business was to remove all of the dumps, zips, and sh*t left-over from developers that had no clue what they were doing. Typically these are all security risks out in the open, and the customer isn’t even aware of it. MySQL files, zipped code, test files, and anything and everything that didn’t belong to Magento or make sense was deleted.
Then we proceeded to upgrade from Magento 2.3.0 OSE to 2.3.5p2 w/ PHP 7.3 and find out what breaks along the way. Sure enough things started to break. Getting the customer to listen to our experience, and act upon it is sometimes very difficult, but with Joe the situation was ideal. He just kept saying just do-it, and took advantage our recommendations.
Remove to Improve!
That’s our mantra. A lot of customers, AS included, didn’t want us to remove their treasure trove of mostly garbage 3rd-party modules that they spent [wasted] thousands of dollars on over the years. Yet, even through customers know they are buggy, are costly, full of security holes, and kill performance, they really don’t know any other way to accomplish those functions.
The aMiSTACX team goes in and guts it all. We disable all modules not in use, remove almost as many 3rd party modules as we can, pull all fonts to the local server, comb through the GTMetrix waterfall reports, and move any type of GA or GA Tag manager to the CDN. [Third-party remote scripts are the #1 performance killer of any e-commerce site.]
We hit one major hurdle with Aussie Strength, they needed to keep HubSpot. When we had HS disabled, load time improved remarkably, but we couldn’t find a way to improve the speed. [Later, post live, we would revisit the HubSpot performance issue, and work a little magic to drop load times.]
Avoiding the Paradox Business Model
What many business owners don’t realize is they put themselves into a Paradox Business Model. They are paying others for Marketing Modules / Analytics Modules or offsite services that use local scripts that actually slow down their site, and as a result in degraded performance actually decrease sales, so then to compensate for the speed loss, the business owner then pays others that sell so-called speed modules to slow their site down even further. In desperation, many hire incompetent developers that then add more modules that don’t improve performance, but cause all types of bugs and UX problems. What an endless nightmare! The Paradox Business Model is the primary reason owners consider moving to even more expensive managed solutions.
Simplicity is Performance
Putting an end to the Paradox Business Model. Now that we cleaned years of dumped files, gutted all of the 3rd-party modules that we could, it was time to fix core bugs. We had issues with the Porto theme as Magento was set up into a multi-view store, when really only single-store mode was needed, we had inventory problems, and checkout problems. On top of those bugs, we kept finding all types of bugs. Some of them even directly related to Magneto 2.3.5p2 Core.
Already Joe was starting to see the difference in speed, and stuff that didn’t ever work, or hadn’t worked in years was now working. It was my call to convince him to keep going. Let’s upgrade to Magento 2.4.0! [As I wanted to take advantage of PHP 7.4 and MySQL 8.]
Now 2.4.0 is a major major Magento upgrade in many ways. Not only does it require PHP 7.4, but it is also 100% integrated with Elasticsearch, and additionally it requires MySQL 8. [Warning: You can NOT downgrade back to 5.7 from MySQL 8. It’s a one way trip.] Elasticsearch is a performance hog, and difficult to maintain. Gone are the days that you can run on anything less than t3-large for any site that gets a decent amount of traffic. Thankfully Joe wanted to leverage Algolia Search, but it still doesn’t remove the ES dependency. ES must be in a running valid state.
This part of the project took a lot of time, because 2.4.0 was only released only about two months ago from the time we started this project, and many vendors were just catching up, Algolia Search included. Many other module vendors were also still very far behind. Our team had to do a few workarounds to get it everything working.
I didn’t know Magento could be this fast!
More More More!
The site was transforming into something unrecognizable to the client. We were now using Algolia Search, Magento 2.4.0, and we started to install aMiSTACX performance modules Ramjet and S3Rocket. Aussie Strength now packed on some serious muscle, and Joe was starting to get anxious to go live. Yet, again, I had to object and explain let’s fix it all, let’s just keep going. [Plus, we were still reasonably withing the client’s budget.]
My reasoning was simple. While the site was in dry dock, let’s attempt to upgrade to Magneto 2.4.1 [It was just released, and fixed a whole bunch of bugs, and offered even more performance.]
The upgrade to 2.4.1 went smoothly; however, there was a slight issue with Porto, but we quickly resolved it with a request for a patch from their support team.
Cleanup and Pre Go-Live
Our final steps were to resolve some major CSS issues, some font changes, set up Trak and NTS, and to implement GIT. Additionally, we had requested numerous QA cycles from the client to catch any bugs that we didn’t want to unexpectedly find after we flipped the switch to go-live. As a clone of Dev would become Prod, and Dev and Prod would be in-sync via GIT. The worst thing you can do is to not properly QA while the server was in Dev dry-dock mode. [We suggested a simple Excel Spreadsheet with items you can checkoff as open, closed, and maybe a pre-existing or new bug field with a simple description.]
Image. aMiSTACX Trak for Matomo
Matomo Analytics w/ aMiSTACX Trak and Nail-the-Sale
It didn’t take much convincing to get AS to opt for our S3 Titanium one-page checkout module called Nail-the-Sale [NTS], and Trak our Matomo Analytics bridge module for Magento. NTS was designed to deliver faster performance through checkout, and also allowed abandoned cart emails. Bridged with Matomo through Trak, we can keep track of conversion rates, abandoned cart revenue, products left in the cart, and so much more. Plus, since everything is designed for maximum speed, there was NO GA to slow the site down, and no data mining private customer data. [Note: A special aMiSTACX server for Magento is required to get Trak to function properly.]
S3 Titanium Performance
We now had Magento running with 2.4.1 OSE taking advantage of the aMiSTACX G5 S3 Titanium LAP stack. Part of our Enterprise Suite of modules was Ramjet cache booster and S3Rocket. WebP was recently introduced for S3Rocket, and it made a considerable difference for Aussiestrength.com.au as their site is very media rich. Ramjet put the site over the top. It really is an amazing cache booster. These working in conjunction with the Cloudflare CDN made performance unbelievable and very stable.
Im getting comments on site speed…
People are saying it’s incredible!
Area 51 Dashboards: Monitor and Manage your Empire
Many of our customers don’t realize the power of Area 51. There is no extra cost to use Area 51 as it is included with every AWS aMiSTACX on Marketplace. It was designed from scratch for speed! NO server side hooks required. It’s also designed to save you money.
Right away, Aussie Strength found better use of A51 after I had spent a few minutes outlining the features. Power control is the #1 cost saver here. And for AS that meant they could create a custom team user account, and have their office staff power EC2s on/off like a light switch, or even via a schedule. Additionally, all the complication of AWS IAM accounts were removed.
Another A51 must-have feature was a way to set up automated backups via a schedule with a retention period. This is extremely important for any e-commerce site.
Joe also went along with my suggestion, and created a one page static maintenance page that we hosted on S3. We then created a Cloudflare rule to flip on/off every time we wanted to direct public traffic to the maintenance page. With Area 51, you also can link your CF profile via API to Area 51. Thus, one dashboard can be used to control the major features of both CF and AWS.
It’s an excellent system!
Extra Security and Performance through Cloudflare
One of the main reasons we only recommend, support, and use Cloudflare is the extra free features such as their Web Access Firewall [WAF]. Part of our remove to improve is also to remove all countries you do not do business with. I had asked Joe to block as many countries as he could as this will help improve performance, and reduce script and malware attacks.
Go-Live and some Unexpected Issues
Mostly, the go-live flip was seamless, and everything worked extremely well. Then all hell broke loose when the Aussies started their day, and started to slam the server with traffic and an email blast. We had been monitoring this t3-xlarge all day, and now we were getting heavy usage beyond normal to the point where 15GB of RAM was quickly consumed, and the server started to crash. It made absolutely no sense as we have load-tested our G4s to higher levels of traffic and utilization should have been < 20% usage.
Right away I though it was bot related or Elasticsearch, so I leveraged Cloudflare’s WAF again to isolate traffic only to Australia. Still the same issue, but thankfully the Apache logs were leaving clues. Memory was exhausted, and appeared due to a Magento function called CSP. Joe was helping us out, and quickly did a search and found an open bug on GitHub that explained that there was a major bug in Magento’s 2.4.0/1 release that would consume huge chunks of memory.
After tuning the server for more than hour to attempt to offset these huge spikes, it was decided to just disable the Magento CSP module as AS had no requirement for it at this time, and the issue was scheduled to be resolved with the future Magento 2.4.2 release. As soon as CSP was disabled, even under heavy traffic, utilization dropped below 10%. That’s the power of caching layers.
On the Fly Updates to Prod
Updates to Prod could be dropped on the fly without much disruption. The workflow and QA process was simple. Development and deploy on Dev server first, sign-off the QA, and then push through GIT. On the Prod side, just pull, and run our recompile script. First flipping the one page maintenance page rule through Area 51. There were several fail-safes along the way should anything unexpected happen. Overall this process worked like a Formula 1 pit crew. Gone are the days of developing, testing, and deploying on a live production e-commerce server.
A Team of Experts at your Side
Feedback from our customers allows continual improvement, and the Aussie Strength project was no different. We caught a bug in Trak that would not list the quantity of items in abandoned carts; that bug was quickly resolved. Feedback about Area 51 from Joe had us improve the UI/UX within minutes or hours. It was like every time Joe went to sleep, and then started his new day it was like Christmas morning. Presents of new features, bugs fixed, and additional performance were all realized within minutes and hours, and not days or weeks like other development teams.
Following up with our client Aussie Strength could not have been more enjoyable. They are super pleased with the speed and stability. It’s hard for even aMiSTACX to believe this is Magento. Aussie Strength avoided the Paradox Business Model, and took our advice remove to improve, and they are reaping the rewards all the way to the $$ bank.
it’s rock solid
[Sales] up about 150% if not more
The aMiSTACX Difference
Overall the project was highly successful. Even though Aussie Strength sold out of most gym inventory due to CoVID-19, post live on the new G5, stability, sales, and conversion all increased. The easy of use and monitoring through A51 allowed more flexibility, and even mobile management. Security increased and more analytical depth through Trak/Matomo was now available.
All you need to succeed:
- aMiSTACX S3 Titanium G5/G4
- aMiSTACX S3Rocket and Ramjet
- aMiSTACX Nail-the-Sale and Trak
- Cloudflare CDN [WAF, Origin SSL Certs]
- Algolia Search
- Matomo Analytics
What you DO NOT need:
- No Redis or any other Dinosaur caching engine
- No NGINX or complicated reverse HTTPS proxy configurations
- Reduction in 3rd party scripts [Stay away from them!]
- No Google Analytics
And to say our favorite tag line doesn’t apply here? Better Stronger Faster 🙂