You've decided the concept makes sense. A tool that reads execution signals from the systems your teams already use — project management, communication, development, calendars — and synthesizes them into an interpreted picture of whether the organization is on track. You understand the value. You've read about what it does and what it doesn't do. You're ready to bring it to your IT team.
This is the moment where many evaluations stall. Not because the product fails the security review, but because the champion — usually the VP of Product or VP of Engineering — doesn't know what questions the review will ask, doesn't know what good answers look like, and can't bridge the gap between "I believe in this" and "here's why our security team should approve it."
This post is for that champion. It is a framework for evaluating any tool that reads organizational signals — structured around the questions your IT lead, CISO, or procurement team will ask, the answers that distinguish a well-architected product from one that's bolted privacy onto a surveillance-era design, and the red flags that should end an evaluation immediately.
The first question: what does it read?
Every tool that provides organizational intelligence reads something from your systems. The evaluation starts with understanding exactly what.
There are two fundamentally different architectures. The first reads content: message bodies, email text, document contents, ticket descriptions, meeting transcripts. The second reads behavioral metadata: communication patterns, task state transitions, coordination structures, delivery cadence — the structural pattern of activity, not the substance of it.
The distinction is not a matter of degree. It is architectural. A system designed to read content has a fundamentally different data model, storage architecture, and risk profile than a system designed to read patterns. The question to ask is not "do you read our data?" — every integration reads data. The question is: what category of data does your system read, and is the boundary between content and metadata enforced at the architecture level or the policy level?
Architecture-level means the system is designed so that content is structurally excluded from persistent storage. The processing pipeline reads metadata fields — timestamps, participants, state changes, frequency counts — and never writes message bodies or document text to any persistent store. If source content is read ephemerally for classification purposes, it is processed in memory and discarded. The structured analytical output is retained. The content is not.
Policy-level means the system could read and store content, but the vendor promises not to. This is a weaker guarantee — because a policy can change, a configuration can be modified, and the underlying architecture permits what the policy prohibits.
Ask: is the content exclusion architectural or policy-based? If the vendor cannot clearly articulate which one, that is a red flag.
The second question: what does it store?
Reading and storing are different operations with different risk profiles. A system that reads behavioral metadata and stores aggregated patterns has a narrower data footprint than one that reads and stores raw content.
The evaluation should map the vendor's storage model explicitly. What data categories are written to persistent storage? What is the retention policy for each category? Are raw source extracts retained, and if so, for how long and in what form? Is individually identifiable data pseudonymized before it enters persistent analytics datasets? Are integration credentials stored separately from analytics data, in a dedicated secret management system?
The answers here should be specific and verifiable. "We take privacy seriously" is not an answer. "Message bodies are never written to persistent storage; behavioral metadata is stored in tenant-isolated, encrypted partitions with AES-256; integration credentials are held in a dedicated Key Vault namespaced by tenant" — that is an answer.
Ask: can you provide a data map showing exactly what is persisted, in what form, and for how long? If the vendor cannot produce this document, the evaluation should not proceed until they can.
The third question: what does it surface?
What a tool displays to its users determines its practical impact on the organization — and its risk to individual employees.
A tool that surfaces individual-level behavioral data — this person sent fourteen messages, this person was active for six hours, this person's "productivity score" dropped — is an employee monitoring tool, regardless of what else it does. The risk profile is fundamentally different from a tool that surfaces team-level and department-level patterns — this team's cross-functional coordination has declined, this initiative's delivery rhythm has shifted, this department's Momentum score has changed.
The distinction matters for privacy, for trust, and for the organizational dynamics the tool creates. Individual-level surfacing incentivizes performance theater: people optimize for the metric, not for the work. Team-level surfacing incentivizes structural awareness: leaders see patterns that indicate organizational health without evaluating individual behavior.
Ask: does the system surface individual-level behavioral data to any user, in any view, under any circumstance? Are minimum group sizes enforced in all aggregated outputs? If the vendor hesitates on either question, probe further.
The fourth question: where does it run?
Deployment architecture determines who controls the data, who can access it, and what happens if the relationship ends.
There are two common models. In the first, the vendor deploys components into your cloud environment — managed applications, relay functions, storage accounts, or processing infrastructure that runs in your Azure, AWS, or GCP subscription. This model gives you more direct control over the infrastructure but also makes you responsible for its security, maintenance, and teardown.
In the second, the vendor operates a fully managed SaaS platform. Nothing is deployed into your environment. All processing, storage, analytics, and serving run on vendor-managed infrastructure. You access the product through a client application — typically a web interface or, in the case of Teams-integrated products, a Teams personal app.
Neither model is inherently superior. But the evaluation questions differ. For a vendor-deployed model: what resources are created in your subscription? Who manages them? What access does the vendor have to your environment? What is the teardown process? For a SaaS model: how is tenant data isolated? What encryption standards are applied at rest and in transit? What access controls govern vendor employee access to production data?
Ask: can you provide an architecture diagram showing exactly where data lives, how it moves, and who has access at each point? This is the document your security team will need to complete the review.
The fifth question: how is tenant data isolated?
For any multi-tenant SaaS product, isolation is the foundational security control. Your data should never be in scope for another tenant's queries, processes, or outputs.
Isolation should be enforced at every layer: identity (your users are resolved to your tenant before any processing begins), API (every request is scoped by tenant before database or analytics access), secret storage (your integration credentials are namespaced separately from other tenants'), analytics pipeline (every processing run receives explicit tenant scope), and serving (the outputs displayed to your users are drawn exclusively from your tenant's data).
Ask: at which layers is tenant isolation enforced? Is isolation enforced at the application layer only, or is it reinforced at the infrastructure layer (separate storage partitions, namespaced secrets, scoped pipeline execution)? If isolation depends entirely on application logic without infrastructure-level reinforcement, the risk surface is wider than it needs to be.
The sixth question: what happens when you leave?
Offboarding reveals more about a vendor's data practices than onboarding does. A vendor that makes it easy to leave is a vendor that has designed its data model with the customer's interests in mind.
The evaluation should cover: can you export your data before deletion? What is included in the export — scored outputs, configuration, workflow records, alert history? What is the timeline for deletion after the contract ends? Is deletion confirmed in writing? Is there any data that persists after deletion (aggregated, anonymized, or otherwise)?
And critically: does offboarding require any infrastructure teardown on your side? If the vendor deployed components into your environment, removal is your responsibility — and incomplete removal leaves artifacts in your subscription. If the vendor operates fully managed SaaS, offboarding should require no action on your side beyond revoking the application's access to your source systems.
Ask: what does offboarding look like, step by step? What data persists after deletion, if any? Can you provide deletion confirmation?
The seventh question: what compliance posture does the vendor hold?
Compliance certifications are trust signals, but they are not pass/fail tests. A vendor can be well-architected and not yet certified. A vendor can hold a certification and still have gaps in their specific data handling model.
The evaluation should assess compliance posture honestly. Is the vendor SOC 2 Type II certified, or building toward certification? If building toward, are the controls mapped and operating in alignment, and is an architecture review available? Does the vendor offer a BAA for HIPAA-applicable contexts? Is a DPA available for GDPR compliance? Does the vendor support data residency requirements?
Ask: what is your current compliance status — not your target, your current status? If the vendor claims a certification, ask for the report. If they are building toward certification, ask for the architecture review. If they cannot provide either, calibrate your risk tolerance accordingly.
Red flags that should end an evaluation
Some answers are disqualifying. If you encounter any of the following, the evaluation should stop:
The vendor cannot clearly articulate the boundary between content and metadata in their data model. The vendor stores message bodies, email content, or document text in persistent storage without a clear, time-bound retention and deletion policy. The vendor surfaces individual-level behavioral data to managers or administrators. The vendor cannot produce a data map, architecture diagram, or tenant isolation description. The vendor's content exclusion is policy-based rather than architecture-based, and they cannot explain why. The vendor has no offboarding process, or the offboarding process does not include data deletion. The vendor claims compliance certifications they cannot substantiate with documentation.
None of these are subjective. They are structural tests. A vendor that passes them has built their product with data handling as a design constraint. A vendor that fails them may still have good intentions — but intentions are not architecture.
The champion's role
The VP who brings this evaluation to their IT team is not expected to be a security expert. They are expected to be a credible advocate — someone who has done enough diligence to present the product seriously and answer the first round of questions before the security team takes over.
This framework gives the champion the language for that conversation. The seven questions map to the concerns IT and procurement will raise. The red flags provide a filter that demonstrates the champion has already screened for the most common failures. And the specificity of the questions — what is read, what is stored, what is surfaced, where it runs, how it's isolated, what happens at offboarding, and what compliance posture exists — signals that the champion is approaching this as a serious evaluation, not a product pitch.
The security team will conduct their own review. The champion's job is to get the product to that review with credibility intact — theirs and the product's.
