Choosing a security awareness training platform is not complicated in theory. But in the real-world, most IT teams end up either paying for a platform that barely gets used, or spending more time managing it than they expected. The market is crowded, the claims are similar, and the differences that matter are not always obvious from a demo.
These 12 questions separate the platforms worth considering from the ones that will create more work than they solve. They are not designed to find the most feature-rich platform. They are designed to find the one that will work for a lean IT team, run without constant attention, and produce measurable behaviour change over time.
1. How much of your time will setup take?
Setup time is a real cost, not a headline metric. Before you evaluate any platform, ask vendors to walk you through every step on your side, not just theirs. Some platforms involve directory integrations to configure, email security settings to modify, and user lists to import manually. Others connect to your existing Microsoft infrastructure and handle the rest automatically. The difference matters before you sign, not after.
Ask: Walk me through what we need to do on our side to get from contract signed to first simulation running.
2. How automated is the ongoing user management?
User management is where hidden admin time is spent. Every time an employee joins, leaves, or changes role, the platform needs to reflect it. Platforms that sync directly with your directory infrastructure handle this automatically. Platforms that rely on manual imports or CSV uploads put that work back on the IT team, and it compounds over time.
Ask: How does the platform handle joiners, leavers, and role changes? What needs to come from us?
3. How much ongoing admin does it create?
There is a meaningful difference between a platform that runs itself and one that requires regular input to function. Campaign scheduling, content management, chasing completions, and compiling compliance reports are all legitimate platform activities but if they sit on your team, they represent a real ongoing cost. For a platform built for lean teams, ongoing involvement should be close to zero once setup is complete.
Ask: What does a typical customer need to do each month? What happens if nobody logs in for 30 days?
4. Do simulations reach the inbox?
Phishing simulations only work if they land in the employee inbox looking and behaving like a real attack. Some platforms require IT teams to whitelist sending domains in email security settings, which means your email gateway treats simulation emails differently from genuine phishing. That undermines the validity of the test and creates a maintenance dependency every time your security configuration changes.
Ask: Do your simulations require any whitelisting or changes to our email security configuration?
5. What attack scenarios do the simulations cover?
Not all phishing looks the same. Invoice fraud, C-suite impersonation, brand spoofing, credential harvesting, and fake security notifications are distinct attack patterns that require employees to recognise different threat signals. A platform with a narrow simulation library will prepare your team for a narrow range of attacks. Look for breadth of scenario coverage, content that reflects current threat techniques, and simulations that escalate in sophistication as employees improve.
Ask: Can you walk me through the range of simulation scenarios available? How does the content evolve as employees progress?
6. Are simulations delivered in your employees' native language?
A simulation delivered in a language an employee does not primarily work in is not a realistic test. For organisations operating across multiple regions, native language delivery is not a nice-to-have - it is the difference between training that feels relevant and training that gets ignored. Check which languages are supported and whether localisation applies to the simulation content itself, not just the platform interface.
Ask: What languages do your simulations support? Is localisation applied to the content of the phishing scenarios or only the platform UI?
7. Continuous or campaign-based?
This is the most important methodological question on this list. Most platforms run on a campaign model where training goes out quarterly or annually and employees complete it. The problem is that this approach often struggles to sustain lasting behaviour change. Evidence suggests that phishing susceptibility often returns to near-baseline within a few months of a one-off training intervention. Short, frequent, contextual touchpoints triggered by behaviour rather than a calendar produce sustained improvement. Before evaluating platforms, decide which outcome you are optimising for.
Ask: Is your platform built around continuous training or scheduled campaigns?
8. Is the training personalised, and does it adapt?
Generic training, with the same module for every employee regardless of role or past performance, is one of the main reasons awareness programmes produce limited results. A finance team member faces different threats to someone in operations. Personalisation means role-relevant content and difficulty levels that adapt based on how each employee is responding. A well-designed platform should increase difficulty as employees improve and increase frequency for those who are struggling, without requiring manual configuration.
Ask: How does the platform adapt frequency and difficulty automatically? Can you show me two different employee experiences?
9. What happens when someone clicks or leaks information?
The moment an employee clicks a simulated phishing link is the highest-impact learning opportunity in the programme. A generic "you've been phished" landing page misses it. The most effective intervention is immediate, specific, and contextual, explaining precisely what the employee should have noticed and what to do differently. An email sent the following day with remedial training has significantly less impact than an in-context lesson triggered at the point of failure.
Ask: Can you show me exactly what an employee sees immediately after clicking? Not a description, the actual thing.
10. How is success measured beyond click rates?
Click rates matter but they are not the whole picture. An organisation where employees actively report suspicious emails they did not click is in a fundamentally different position from one where click rates are low but nobody is flagging anything. The metrics worth tracking over time include reporting rate, repeat failure rate, engagement rate, difficulty distribution, and difficulty progression. If a platform can only show you click rates, it is measuring one dimension of a multi-dimensional problem.
Ask: Beyond click rates, what metrics do you track? Can you show me an example report at 6 and 12 months?
11. What does compliance reporting look like?
If you are working towards ISO 27001, NIS2, or Cyber Essentials, or need to evidence a training programme to your cyber insurer, compliance reporting is not an add-on. It is part of what you are buying. Ask vendors to generate a compliance report live in the demo. Note how many steps it takes, whether it requires manual configuration, and whether the output is in a format an auditor would accept without additional documentation.
Ask: Can you generate a compliance report and show me what it contains?
12. Are you paying for features you will never use?
Many security awareness platforms are built for enterprise procurement cycles, modular and configurable and priced accordingly. For an SMB or mid-market IT team, that often means paying for complexity you will never use. The right platform for a 200-person organisation with a two-person IT team covers the essentials without requiring a procurement exercise to access them. Simulations, training, compliance reporting, and analytics should all be standard, not add-ons behind a higher tier.
Ask: What is included in this price, and what would I need to upgrade to get compliance reporting, adaptive simulations, and behavioural analytics?
One final note
The best way to answer most of these questions is to see the platform running in your own environment. Any vendor worth considering should be able to demo using your domain, not a pre-configured sandbox. Most offer a free trial. Use it.
If you’re evaluating security awareness training
Pistachio Practice is built specifically for lean IT teams in SMB and mid-market organisations. It deploys in around 10 minutes via Microsoft, syncs automatically with Entra ID, and runs continuously from there - no campaigns to schedule, no employees to chase, no ongoing configuration required.
Simulations require no whitelisting or email security changes and are drawn from a library of over 4,000 scenarios covering invoice fraud, C-suite impersonation, brand spoofing, credential harvesting, and more. Training adapts automatically to each employee's role and performance. Compliance reports for ISO 27001, NIS2, and Cyber Essentials are one click.
"Traditional phishing training feels like a box-ticking exercise, but Pistachio builds real skills over time. My users enjoy the tests, get instant feedback, and even brag about spotting fake emails. I can see a clear improvement in awareness and vigilance that lasts all year, not just after a one-off training session." — Joseph Cunningham, Head of IT Security & Support, OOONO
Email contact@pistachioapp.comto discuss the platform with a member of our team.
