Research Library / Case Studies /

MedTech / Healthcare AI

LLM Security Assessments in AI- Powered MedTech Applications

A MedTech leader's AI chat assistant could be jailbroken into giving medical advice it was explicitly told not to give. Payatu tested it against 13 LLM attack classes before real patients ever saw a response.

LLM Security Assessment, Mobile Application Security Testing (iOS), AI/ML Security Audit

At a glance

INDUSTRY

MedTech / Healthcare AI

CLIENT PROFILE

A renowned MedTech leader offering AI-powered chat support alongside its medical and surgical equipment

SERVICES

LLM Security Assessment, Mobile Application Security Testing (iOS), AI/ML Security Audit

ENGAGEMENT

Web LLM application and companion iOS app

Key Takeaways

Client – a renowned MedTech leader that built a chat-based application using LLM models to answer patient and customer prompts

Problem – needed to identify security flaws in both the LLM model and its mobile application before customers used it consistently, given the sensitivity of medical guidance.

What Payatu did – ran a dedicated LLM security assessment covering prompt injection, jailbreaking, and system prompt leakage, alongside a full iOS application security assessment.

Outcome – found that the model could be jailbroken into bypassing its own medical-advice restrictions, plus system prompt leakage and mobile application gaps, each mapped to explicit remediation guidance.

the challenge

Why the client called us in

Before customers could rely on it consistently, the client needed to know whether its LLM-powered chat assistant could be manipulated into unsafe or incorrect behaviour, and whether its companion mobile application held up to the same scrutiny as any other patient-facing product. Payatu was brought in to identify the flaws, explain the risk, and guide remediation priority.

Test the LLM for prompt injection, jailbreaking, and system prompt leakage
Assess the iOS application across static, dynamic, and runtime protection layers
Prioritise findings by real clinical and business risk, not just technical severity

scope of engagement

What was in scope

LLM security assessment on the web application
Mobile (iOS) application security assessment

Our Approach

How Payatu ran the engagement

01

LLM Architecture and Access Mapping

Identified the underlying LLM architecture, reviewed model documentation, and mapped access methods, authentication mechanisms, and user permissions.

02

Attack Surface Enumeration

Built a list of known AI attack classes, including prompt injection, jailbreaking, and system prompt leakage, tailored to the client's chat application.

03

Model Testing

Tested the model for biased or harmful responses and attempted to bypass content moderation filters using altered phrasing and encoding.

04

iOS Application Testing

Ran static analysis, reverse engineering, local file analysis, and dynamic analysis on the iOS application, covering binary protections, local authentication, and insecure data storage.

05

Documentation and Reporting

Documented every vulnerability with reproduction steps and CVSS v3.1 scoring, then reported findings with prioritised remediation guidance.

Key findings

What we found

HIGH

Model jailbreaking bypassed medical-advice restrictions

The system prompt instructed the model not to give medical advice, but after a jailbreak it suggested medicines and treatments regardless.

HIGH

Misinformation and malicious content generation

The model could be manipulated into generating and reinforcing false information, and into producing malicious content at the request of an attacker.

the outcome

Results and Impact

Payatu's assessment gave the client a clear, risk-ranked view of how its AI assistant could fail before patients ever depended on it, closing the gap between a working demo and a product safe for clinical-adjacent use.

‍