Developer Guide

Contents:

• Introduction
  • Intended Audience
  • Getting Help
  • Core Technologies
  • Roadmap
  • Kanban Board
  • Issue Tracker
  • Related Guides
  • Related Projects
• Development Environment
  • Quickstart
  • Detailed Steps
    • Install Java
    • Install Maven
    • Install and Start Docker
    • Git Clone Repo
    • Build and Run
    • Verify
  • Next Steps
  • Getting Help
• Windows Development
  • Running Dataverse in Docker on Windows
  • Running Dataverse in Windows WSL
    • Install WSL
    • Install Dataverse
    • IDE for Dataverse in Windows
    • pgAdmin in Windows for Dataverse
• Tips
  • Iterating on Code and Redeploying
    • Undeploy the war File from the Dataverse Software Installation Script
    • Add Payara as a Server in Netbeans
    • Ensure that the Dataverse Software Will Be Deployed to Payara
    • Make a Small Change to the Code
    • Confirm the Change Was Deployed
  • Netbeans Connector Chrome Extension
  • Thumbnails
  • Database Schema Exploration
    • Database Name and Credentials
    • pgAdmin
    • SchemaSpy
  • Deploying With asadmin
  • Running the Dataverse Software Installation Script in Non-Interactive Mode
  • Preventing Payara from Phoning Home
  • Solr
  • Git
    • Set Up SSH Keys
    • Git on Mac
    • Automation of Custom Build Number on Webpage
  • Sample Data
  • Switching from Glassfish to Payara
  • UI Pages Development
    • Avoiding Inefficiencies in JSF Render Logic
• Troubleshooting
  • context-root in glassfish-web.xml Munged by Netbeans
  • Configuring / Troubleshooting Mail Host
  • Rebuilding Your Dev Environment
  • DataCite
• Version Control
  • Where to Find the Dataverse Software Code
  • Branching Strategy
    • Goals
    • Branches
      • The “master” Branch
      • The “develop” Branch
      • Feature Branches
      • Hotfix Branches
  • How to Make a Pull Request
    • Find or Create a GitHub Issue
      • Finding GitHub Issues to Work On
      • Creating GitHub Issues to Work On
    • Communicate Which Issue You Are Working On
    • Create a New Branch Off the develop Branch
    • Commit Your Change to Your New Branch
    • Writing a Release Note Snippet
    • Push Your Branch to GitHub
    • Make a Pull Request
    • Replace Issue with Pull Request
    • Make Sure Your Pull Request Has Been Advanced to Code Review
    • Summary of Git commands
  • How to Resolve Conflicts in Your Pull Request
  • Adding Commits to a Pull Request from a Fork
  • Handing a Pull Request from a “develop” Branch
• SQL Upgrade Scripts
  • Location of SQL Upgrade Scripts
  • How to Determine if You Need to Create a SQL Upgrade Script
  • How to Create a SQL Upgrade Script
  • Troubleshooting
    • Renaming SQL Upgrade Scripts
• Testing
  • The Health of a Codebase
  • Testing in Depth
  • Unit Tests
    • Unit Test Automation Overview
    • Writing Unit Tests with JUnit
      • Refactoring Code to Make It Unit-Testable
      • Parameterized Tests
      • JUnit 5 Test Helper Extensions
      • Observing Changes to Code Coverage
      • Testing Commands
      • Running Non-Essential (Excluded) Unit Tests
  • Integration Tests
    • Running the Full API Test Suite Using EC2
    • Running the Full API Test Suite Using Docker
    • Running the APIs Without Docker (Classic Dev Env)
      • The Burrito Key
      • Root Collection Permissions
      • Publish Root Collection
      • dataverse.siteUrl
      • dataverse.oai.server.maxidentifiers
      • dataverse.oai.server.maxrecords
      • Identifier Generation
    • Writing API Tests with REST Assured
    • Writing and Using a Testcontainers Test
  • Measuring Coverage of API Tests
    • Add jacocoagent.jar to Payara
    • Add jacococli.jar to the WAR File
    • Deploy the Instrumented WAR File
    • Run API Tests to Determine Code Coverage
    • Create Code Coverage Report
    • Read Code Coverage Report
  • Load/Performance Testing
    • Locust
    • download-files.sh script
  • Continuous Integration
    • Enhance build time by caching dependencies
    • The Phoenix Server
      • How the Phoenix Tests Work
      • How to Run the Phoenix Tests
  • Accessibility Testing
    • Accessibility Policy
    • Accessibility Tools
  • Future Work
    • Future Work on Unit Tests
    • Future Work on Integration Tests
    • Browser-Based Testing
    • Installation Testing
    • Future Work on Load/Performance Testing
    • Future Work on Accessibility Testing
• API Design
  • OpenAPI
  • Paths
    • Exposing Settings
• Security
  • Intake of Security Issues
  • Sending Security Notices
• Performance
  • Problem Statement
  • Current Practices
    • Cache When You Can
    • Use Async
    • Use a Queue
    • Offload Expensive Operations Outside the App Server
    • Drop to Raw SQL as Necessary
    • Add Indexes to Database Tables
    • Find Bottlenecks with a Profiler
    • Warn Developers in Code Comments
    • Write Docs for Devs about Perf
    • Horizontal Scaling of App Server
    • Code Review and QA
    • Locust Testing at Release Time
    • Issue Tracking and Prioritization
    • Document Performance Tools
    • Google Analytics
  • Abandoned Tools and Practices
    • New Relic
  • Areas of Particular Concern
    • Command Engine Execution Rate Metering
    • Solr
    • Datasets with Large Numbers of Files or Versions
    • Withstanding Bots
  • Suggested Practices
    • Implement the Frontend Plan for Performance
    • Set up Query Counter in Jenkins
    • Count Database Queries per API Test
    • Teach Developers How to Do Performance Testing Locally
    • Automate Performance Testing
    • Make Production Data or Equivalent Available to Developers
    • Automate Performance Testing with Production Data or Equivalent
    • Use Monitoring as Performance Testing
    • Learn from Training and Conferences
    • Learn from the Community How They Monitor Performance
    • Teach the Community to Do Performance Testing
  • Conclusion
• Dependency Management
  • Introduction
  • Terms
  • Direct dependencies
  • Transitive dependencies
    • Managing transitive dependencies in pom.xml
  • Helpful tools
  • Repositories
  • Dataverse Parent POM
• Debugging
  • Logging
  • Java Server Faces (JSF) Configuration Options
• Coding Style
  • Java
    • Formatting Code
      • Tabs vs. Spaces
      • Imports
      • Braces Placement
      • Format Code You Changed with Netbeans
      • Checking Your Formatting With Checkstyle
    • Logging
    • Avoid Hard-Coding Strings (Use Constants)
    • Avoid Hard-Coding User-Facing Messaging in English
    • Type Safety
  • Bash
    • Formatting Code
      • Tabs vs. Spaces
  • Bike Shedding
• Consuming Configuration
  • Simple Configuration Options
  • Complex Configuration Options
  • Why should I care about MicroProfile Config API?
  • Adopting MicroProfile Config API
    • Adding a JVM Setting
    • Moving or Replacing a JVM Setting
    • Adding a Feature Flag
• Deployment
  • Deploying the Dataverse Software to Amazon Web Services (AWS)
    • Install AWS CLI
      • Troubleshooting “aws: command not found”
    • Configure AWS CLI with Stored Credentials
    • Configure Role-Based S3 Access
    • Configure Ansible File (Optional)
    • Download and Run the “Create Instance” Script
    • Caveat Recipiens
    • Migrating Datafiles from Local Storage to S3
      • To Update Dataset Location to S3, Assuming a file:// Prefix
      • To Update Datafile Location to your-s3-bucket, Assuming a file:// Prefix
      • To Update Datafile Location to your-s3-bucket, Assuming no file:// Prefix
• Docker, Kubernetes, and Containers
  • Container Guide
  • Help Containerize Dataverse
  • Community-Lead Projects
  • Using Containers for Reproducible Research
• Making Releases
  • Introduction
  • Regular or Hotfix?
  • Ensure Issues Have Been Created
  • Declare a Code Freeze
  • Conduct Performance Testing
  • Conduct Smoke Testing
  • Write Release Notes
  • Upgrade Instructions for Internal
  • Deploy Release Candidate to Demo
  • Prepare Release Branch
  • Merge “develop” into “master”
  • Add Milestone to Pull Requests and Issues
  • (Optional) Test Docker Images
  • Build the Guides for the Release
  • Create a Draft Release on GitHub
  • Run a Build to Create the War File
  • Build Installer (dvinstall.zip)
  • Make Artifacts Available for Download
  • Publish the Release
  • Update Guides Link
  • Close Milestone on GitHub and Create a New One
  • Update the Container Base Image Version Property
  • Deploy Final Release on Demo
  • Update SchemaSpy
  • Alert Translators About the New Release
  • Add the Release to the Dataverse Roadmap
  • Announce the Release on the Dataverse Blog
  • Announce the Release on the Mailing List
  • For Hotfixes, Merge Hotfix Branch into “develop” and Rename SQL Scripts
• Making Library Releases
  • Introduction
  • Maven Central (Java)
    • Releasing Existing Libraries to Maven Central
      • Releasing a Snapshot Version to Maven Central
      • Releasing a Release (Non-Snapshot) Version to Maven Central
    • Releasing a New Library to Maven Central
      • Updating pom.xml for a Snapshot Release
      • Configuring Secrets
  • npm (JavaScript/TypeScript)
• Metadata Export Formats
  • Introduction
  • Exporter Basics
  • Building an Exporter
  • Specifying a Prerequisite Export
• Tools
  • Tools for Faster Deployment
  • Netbeans Connector Chrome Extension
  • pgAdmin
  • Maven
  • PlantUML
  • Eclipse Memory Analyzer Tool (MAT)
  • PageKite
  • MSV
  • FontCustom
  • SonarQube
  • Infer
  • lsof
  • JProfiler
  • jmap and jstat
• Universal Numerical Fingerprint (UNF)
  • UNF Version 3
  • UNF Version 5
  • UNF Version 6
    • I. UNF of a Data Vector
    • II. Combining multiple UNFs to create UNFs of higher-level objects.
    • Footnotes:
• Make Data Count
  • Architecture
  • Dev Environment Setup for Make Data Count
    • Generate Fake Metrics Only
    • Full Setup
  • Testing Make Data Count and Your Dataverse Installation
  • Processing Archived Logs
  • Resources
• Shibboleth, OAuth and OIDC
  • Shibboleth and OAuth
  • OpenID Connect (OIDC)
• Geospatial Data
  • How The Dataverse Software Ingests Shapefiles
    • Ingest
    • Example
• SELinux
  • Introduction
  • Development Environment
  • Recreating the shibboleth.te File
    • Ensure that SELinux is Enforcing
    • Removing the Existing shibboleth.te Rules
    • Exercising SELinux denials
    • Stub out the new shibboleth.te file
    • Iteratively Use audit2allow to Add Rules and Test Your Change
• Big Data Support
  • S3 Direct Upload and Download
    • Features that are Disabled if S3 Direct Upload is Enabled
    • Allow CORS for S3 Buckets
    • S3 Tags and Direct Upload
  • Trusted Remote Storage with the remote Store Type
  • Globus File Transfer
• Auxiliary File Support
  • Adding an Auxiliary File to a Datafile
  • Downloading an Auxiliary File that Belongs to a Datafile
  • Listing Auxiliary Files for a Datafile by Origin
  • Deleting an Auxiliary File that Belongs to a Datafile
• Direct DataFile Upload/Replace API
  • Overview
  • Requesting Direct Upload of a DataFile
  • Upload Files to S3
  • Adding the Uploaded File to the Dataset
  • To Add Multiple Uploaded Files to the Dataset
  • Replacing an Existing File in the Dataset
  • Replacing Multiple Existing Files in the Dataset
• Globus Transfer API
  • Requesting Upload or Download Parameters
  • Performing an Upload/Transfer In
  • Adding Files to the Dataset
  • Downloading/Transfer Out Via Globus
• Dataset Semantic Metadata API
  • Get Dataset Metadata
  • Add Dataset Metadata
  • Delete Dataset Metadata
  • Create a Dataset
  • Create a Dataset with a Dataset Type
• Dataset Migration API
  • Start Migrating a Dataset into a Dataverse Collection
  • Publish a Migrated Dataset
• Workflows
  • Introduction
    • Administration
    • Available Steps
      • log
      • pause
      • pause/message
      • http/sr
      • http/authext
      • archiver
      • ldnannounce
• Font Custom
  • Previewing Icons
  • Install Font Custom
    • Ruby Version
    • Install Dependencies and Font Custom Gem
  • Updating Icons
    • Updating the Guides Icons
    • Updating the App Icons
    • Committing Changes to Git
  • Caveats About Font Custom
• Classic Dev Environment
  • Quick Start (Docker)
  • Classic Dev Environment
  • Set Up Dependencies
    • Supported Operating Systems
    • Install Java
    • Install Netbeans or Maven
    • Install Homebrew (Mac Only)
    • Clone the Dataverse Software Git Repo
    • Build the Dataverse Software War File
    • Install jq
    • Install Payara
    • Install Service Dependencies Directly on localhost
      • Install PostgreSQL
      • Install Solr
    • Install Service Dependencies Using Docker Compose
  • Run the Dataverse Software Installer Script
    • Verify the Dataverse Software is Running
  • Configure Your Development Environment for Publishing
  • Configure Your Development Environment for GUI Edits
  • Next Steps