Pack makes it easy to deliver flicker-free A/B test experiences on landing pages or your Hydrogen storefront. Instead of replacing your data and analytics platforms, Pack works alongside your existing tech stack to deliver the experience side of testing while your analytics tools handle measurement. Let's explore what you can do with Pack's A/B testing feature, and how you can set it up for success.
Prerequisites: Is your data foundation ready?
Before you launch your first A/B test experience on Pack, you'll need solid data infrastructure in place. Pack handles A/B testing experience delivery—your analytics stack handles measurement.
Pack works best when you have:
- Clean, standardized data collection via partners like Elevar, Fueled, Blotout, or Littledata
- Data warehouse infrastructure (BigQuery, Snowflake, Databricks, etc.)
- Established analytics governance and clear KPI definitions
- Server-side event tracking with accurate attribution
Signs you might need to build your data foundation first:
- No existing data infrastructure or unclear measurement strategy
- Expecting Pack to resolve analytics discrepancies or own attribution
- Looking for Pack to replace your analytics stack
Don't worry if you're not there yet! Pack's team works with data partners regularly and can help you evaluate your setup and connect you with the right tools to get your foundation ready. Book some time with us to discuss your specific situation.
Set your A/B test experiences up for success
Once your data foundation is solid, here are our recommendations and best-practices to make your test experiences as successful as possible.
Step 1: Understand what you can test with Pack
Pack makes it easy to deliver:
- Content changes—ex. Custom layouts, imagery, text etc. (anything that you can control in the Pack Customizer)
- 3rd party apps or built-in site features—ex. What's the impact of providing product recommendations on our product page?
- Cart elements—ex. What's the impact of having a package protection widget at checkout?
- Code-based changes—ex. Webhooks and packages so you can create your own unique test experiences
Pack's biggest strengths:
Out of the box, Pack's optimized for:
- Super-fast page loads during tests
- Dynamic content delivery
- No test variant flickering
- Minimizing privacy regulation
That means they're going to give you cleaner test experiences than most platforms, since the shopper experience is top-notch. Pack can deliver this smooth experience because it can fully compile a test experience before it even hits your shopper's device (that's thanks to server-side rendering—the new technology that powers Shopify Hydrogen!).
Pack's testing is particularly powerful and useful when you're experimenting with personalization. Soon you'll be able to use data from your existing "CDP" or anywhere you have access to your shoppers information or user behavior (I.e. location, page visits, segment IDs, etc) to change content dynamically on the page without requiring the shopper to do a refresh to see the personalized content. Because all the experience data is compiled on the server, it can also help avoid some of the client-side data privacy challenges.
When Pack won't work for you:
Pack does not enable tests for data that is not stored within Pack (ex. product pricing), which means you'll want to integrate other tools for more intricate or complex tests (ex. Multi-step content activation).
Step 2: Establish Your Testing Strategy
Before you deliver any test experiences, you'll want to clearly define your:
- Process or framework for ideating and prioritizing high-impact optimizations (Ex: scoring ideas based on reach, impact, confidence, effort (RICE))
- Business objective (Ex: Boost AOV)
- Hypothesis (Ex: by adding a product recommendations slider to the product page, we can boost AOV x%)
- Key metrics (Ex: AOV with do-no-harm for conversion rate)
- Plan for data analysis and interpretation (Ex: Fueled for data collection, Blotout for analysis)
- Process for documenting and sharing learnings
With a documented testing strategy, you're much more likely to end up with test experiences that deliver both accurate and actionable results.
Step 3: Connect Pack to your analytics stack
Pack delivers A/B test experiences and surfaces event data that your existing analytics platforms consume for measurement. Pack's dashboard provides baseline insights like total users, engagement metrics, CVR, AOV, and revenue for your control and variant groups.
To connect your data: today you can integrate with Google Analytics and/or BigQuery, and Pack uses Bayesian statistics to calculate a winner. For more advanced analysis—like custom confidence levels, Frequentist testing, or complex event funnels—export your results to your data warehouse and visualization tools.
Quick tip: If you don't have historical benchmarks, run an A/A test first to establish a baseline for interpreting future results.
Step 4: Set up your first test experience
To start delivering your first A/B test experience, you'll want to pull in a developer / Pack's support team to help you install some packages on your storefront (see our A/B testing developer documentation which covers all the technical prerequisites and setup steps in detail).
Once you've run through setup, it's easy to create your test experience in Pack's visual editor and analyze the results through your connected analytics platforms: