{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Bala Priya C",
  "title_listed": "PrivacyRaven: Comprehensive Privacy Testing for Deep Learning",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/balapriya/interview-with-patrick-cason-engineering-team-lead-at-openmined-1ojb",
  "publish_date": "2021-05-17",
  "corpus_stage": "",
  "notes": "",
  "id": "8f3a9b759d2a",
  "collected_at": "2026-10-02T14:01:32",
  "title": "PrivacyRaven: Comprehensive Privacy Testing for Deep Learning - DEV Community",
  "body_markdown": "This is a summary of Suha S. Hussain’s talk at the OpenMined Privacy Conference 2020 on PrivacyRaven - a comprehensive testing framework for simulating privacy attacks.\nWhy is privacy a concern?\nToday, deep learning systems are widely used in facial recognition, medical diagnosis and a whole wealth of other applications. These systems, however, are also susceptible to privacy attacks that can compromise the confidentiality of data which, particularly in sensitive use cases like medical diagnosis, could be detrimental and unethical.\nDoes a restricted setting necessarily ensure privacy?\nConsider a medical diagnosis system using a deep learning model to detect brain-bleeds from images of brain scans, as shown in the diagram below. This is a binary classification problem, where the model only outputs either a Yes(1) or No(0).\nMedical diagnosis system to detect brain-bleed (Image Credits: Suha S. Hussain)\nGiven that an adversary has access only to the output labels, doesn’t it seem too restrictive a system for the adversary to learn anything meaningful about the model? Well, before we answer this question, let’s understand what an adversary modeled by PrivacyRaven could learn by launching attacks on such a restrictive system.\nExploring PrivacyRaven’s capabilities\nAll attacks that PrivacyRaven launches are label-only black-box - which essentially means that an adversary can access only the labels, not the underlying model’s parameters.\nDifferent privacy attacks that an adversary can launch (Image Credits: Suha Hussain)\nBy launching a model extraction attack, the adversary can steal the intellectual property by successfully creating a substitute model.\nBy launching a model inversion attack, the adversary can reconstruct the images used to train the deep learning images.\nBy launching a membership inference attack,  the adversary can re-identify patients within the training data.\nThreat model\nIn the threat model used by PrivacyRaven, the adversary only receives labels from an API that queries the deep learning model and does not directly interact with the deep learning model at any point.\nThreat model used by PrivacyRaven (Image Credits: Suha S. Hussain)\nPrivacyRaven has been optimized for usability, flexibility and efficiency; has been designed for operation under the most restrictive cases and can be of great help in the following:\nDetermining the susceptibility of the model to different privacy attacks\nEvaluating privacy preserving machine learning techniques\nDeveloping novel privacy metrics and attacks\nRepurposing attacks for data provenance auditing and other use cases\nIn the next section, let’s summarize Model Extraction, Model Inversion and Membership Inference attacks that adversaries modelled by PrivacyRaven can launch.\nModel extraction\nModel extraction attacks are aimed at creating a substitute model of the target system and can be of two types: optimizing for high accuracy or optimizing for high fidelity.\nHigh accuracy attacks are usually financially motivated, such as getting monetary benefits by using the extracted model or avoiding paying for the target model in the future.\nAn adversary optimizing for high fidelity is motivated to learn more about the target model, and in turn, the model extracted from such attacks can be used to launch additional attacks for membership inference and model inversion.\nPrivacyRaven partitions model extraction into multiple phases, namely Synthesis, Training and Retraining.\nPhases in model extraction attack (Image Credits: Suha S. Hussain)\nIn the synthesis phase, synthetic data is generated by using publicly available data, gathering adversarial examples and related techniques.\nIn the training phase, a preliminary substitute model is trained on the synthetic data.\nIn the retraining phase, the substitute model is retrained for optimizing the data quality and attack performance.\nThis modularity of the different phases in model extraction, facilitates experimenting with different strategies for each phase separately.\nHere’s a simple example where, after necessary modules have already been imported, a query function is created for a PyTorch Lightning model included within the library; the target model is a fully connected neural network trained on the MNIST dataset.The EMNIST dataset is downloaded to seed the attack. In this particular example, the ‘copycat’ synthesizer helps train the\nImageNetTransferLearning\nclassifier.\nmodel\n=\ntrain_mnist_victim\n()\ndef\nquery_mnist\n(\ninput_data\n):\nreturn\nget_target\n(\nmodel\n,\ninput_data\n)\nemnist_train\n,\nemnist_test\n=\nget_emnist_data\n()\nattack\n=\nModelExtractionAttack\n(\nquery_mnist\n,\n100\n,\n(\n1\n,\n28\n,\n28\n,\n1\n),\n10\n,\n(\n1\n,\n3\n,\n28\n,\n28\n),\n\"\ncopycat\n\"\n,\nImagenetTransferLearning\n,\n1000\n,\nemnist_train\n,\nemnist_test\n)\nEnter fullscreen mode\nExit fullscreen mode\nThe results of model extraction include statistics of the target model and substitute model, details of the synthetic data, accuracy, and fidelity metrics.\nMembership inference\nIn sensitive applications such as medical diagnosis systems where the confidentiality of patients’ data is extremely important, if an adversary launching a re-identification attack is successful, wouldn’t it sabotage the trustworthiness of the entire system?\nPrivacy concerns in sensitive applications (Image Credits: Suha S. Hussain)\nSimilar to model extraction attacks, membership inference attacks can as well be partitioned into multiple phases in PrivacyRaven. For instance, a model extraction attack is launched to train an attack network to determine if a particular data point is included in the training data, whilst combining it with adversarial robustness calculations. When the adversary succeeds in the membership inference attack, the trustworthiness of the system is indeed sabotaged.\nPhases in membership inference attacks (Image Credits: Suha S. Hussain)\nModel Inversion is the capability of the adversary to act as an inverse to the target model, aiming at reconstructing the inputs that the target had memorized. This would be incorporated in greater detail in future releases of PrivacyRaven.\nFuture directions\nThe following are some of the features that would soon be included in future releases:\nNew interface for metric visualizations.\nAutomated hyperparameter optimization.\nVerifiable Differential Privacy.\nIncorporating attacks that specifically target federated learning and generative models.\nReferences\n[1] GitHub repo of PrivacyRaven:\nhttp://github.com/trailofbits/PrivacyRaven\nCover Image: Photo by\nStefan Steinbauer\non\nUnsplash",
  "metadata": {
    "title": "PrivacyRaven: Comprehensive Privacy Testing for Deep Learning - DEV Community",
    "description": "This is a summary of Suha S. Hussain’s talk at the OpenMined Privacy Conference 2020 on PrivacyRaven... Tagged with research, deeplearning, privacy, security.",
    "keywords": "research, deeplearning, privacy, security, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/balapriya/interview-with-patrick-cason-engineering-team-lead-at-openmined-1ojb",
    "og:title": "PrivacyRaven: Comprehensive Privacy Testing for Deep Learning",
    "og:description": "This is a summary of Suha S. Hussain’s talk at the OpenMined Privacy Conference 2020 on PrivacyRaven...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0oxoot6t9ssyx01up4jy.jpg",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "PrivacyRaven: Comprehensive Privacy Testing for Deep Learning",
    "twitter:description": "This is a summary of Suha S. Hussain’s talk at the OpenMined Privacy Conference 2020 on PrivacyRaven...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F0oxoot6t9ssyx01up4jy.jpg"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/8f3a9b759d2a/code/body-000.txt",
      "chars": 401,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fbs.jpg",
      "bytes": 7062,
      "reused": true,
      "local_path": "attachments/8f3a9b759d2a/images/002.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fbs.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2Fsize%2Fw1000%2F2020%2F12%2Fip.jpg",
      "bytes": 7062,
      "reused": true,
      "local_path": "attachments/8f3a9b759d2a/images/003.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2Fsize%2Fw1000%2F2020%2F12%2Fip.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fthreat-model.png",
      "bytes": 7062,
      "reused": true,
      "local_path": "attachments/8f3a9b759d2a/images/004.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fthreat-model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fmodel-extraction.png",
      "bytes": 7062,
      "reused": true,
      "local_path": "attachments/8f3a9b759d2a/images/005.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fmodel-extraction.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fmembership.jpg",
      "bytes": 7062,
      "reused": true,
      "local_path": "attachments/8f3a9b759d2a/images/006.jpg",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fmembership.jpg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fmembership-inference.png",
      "bytes": 7062,
      "reused": true,
      "local_path": "attachments/8f3a9b759d2a/images/007.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fmembership-inference.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fbs.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2Fsize%2Fw1000%2F2020%2F12%2Fip.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fthreat-model.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fmodel-extraction.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fmembership.jpg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fblog.openmined.org%2Fcontent%2Fimages%2F2020%2F12%2Fmembership-inference.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "http://github.com/trailofbits/PrivacyRaven",
      "label": "http://github.com/trailofbits/PrivacyRaven",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/@usinglight?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Stefan Steinbauer",
      "type": "a"
    },
    {
      "href": "https://unsplash.com/s/photos/secret?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText",
      "label": "Unsplash",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:49:10Z",
  "source_url": "https://dev.to/balapriya/interview-with-patrick-cason-engineering-team-lead-at-openmined-1ojb",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 6542,
  "expected_body_images": 6,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/8f3a9b759d2a/pages/source.html",
  "local_html": "attachments/8f3a9b759d2a/pages/article.html",
  "body_base_path": "attachments/8f3a9b759d2a/pages/",
  "body_markdown_path": "attachments/8f3a9b759d2a/pages/article.md"
}