PRIMARY OBJECTIVE:
I. To compare invasive disease-free survival (IDFS) of patients with triple-negative (TNBC) or low estrogen receptor (ER)-positive and/or HER2 borderline breast cancer who have \>= 1 cm residual invasive breast cancer and/or positive lymph nodes (ypN1mi, ypN1, ypN2, ypN3) after neoadjuvant chemotherapy randomized to receive 1 year of MK-3475 (pembrolizumab) adjuvant therapy compared to no MK-3475 (pembrolizumab), in both the entire study population and also in the PD-L1 positive subset.
SECONDARY OBJECTIVES:
I. To compare the effects of MK-3475 (pembrolizumab) on overall survival (OS) and distant recurrence-free survival (DRFS) between the two randomized arms for the PD-L1 positive patients and then all patients.
II. To assess the toxicity and tolerability of MK-3475 (pembrolizumab) in this patient population with or without radiation therapy.
BEHAVIORAL AND HEALTH OUTCOMES (BAHO) STUDY OBJECTIVES:
I. To examine the association between biomarkers of inflammation and quality of life and patient reported outcomes between the two groups during and at the end of therapy.
II. To examine the long-term and late effects of treatment on patient-reported outcomes.
ADDITIONAL OBJECTIVE:
I. To collect tissue and whole blood for processing and banking in anticipation of future correlative studies in this patient population.
OUTLINE: Patients are randomized to 1 of 2 arms.
ARM I (OBSERVATION): Patients receive no treatment but are monitored at standard clinical intervals during first year after randomization. Patients are examined every 12 weeks for 1 year, and every 6 months for 4 years, then annually for 5 years.
ARM II (PEMBROLIZUMAB): Patients receive pembrolizumab intravenously (IV) over 30 minutes on days 1 and 22. Cycles repeat every 42 days for 52 weeks in the absence of disease progression or unacceptable toxicity.
All patients may undergo radiation therapy within 12 weeks of last breast cancer operation or after treatment. Patients may also undergo collection of blood samples throughout the trial.
After completion of study treatment, patients are followed up every 6 months for 5 years, then annually thereafter until 10 years from treatment randomization.
Inclusion Criteria:
* STEP 1 REGISTRATION
* Patients must have histologically confirmed estrogen receptor (ER)-, progesterone receptor (PR)- and HER2-negative (triple-negative, TNBC) or ER-, PR- weakly positive and/or HER2- equivocal status and must not have received and not be planning to receive adjuvant anti-HER2 or endocrine therapies after completion of neoadjuvant chemotherapy; patients who are HER2-positive by American Society of Clinical Oncology (ASCO) College of American Pathologists (CAP) guidelines are ineligible; HER2-negative and HER2-equivocal cases as per ASCO CAP guidelines that do not receive HER2-targeted therapy are eligible; patients with weakly ER or PR positive disease, defined as ER and/or PR less than or equal to (=\<) 5% by immunohistochemistry, are eligible if the treating physician considers the patient not eligible for adjuvant endocrine therapy; residual disease must be \>= 1 cm in greatest dimension, and/or have positive lymph nodes (ypN1mi, ypN1, ypN2, ypN3) observed on pathologic exam
* NOTE: If the ER and/or HER2 results are discordant between the initial, pre-chemotherapy, and post-chemotherapy surgical tissue, the receptor status of the residual disease has to be used to determine eligibility. Immunohistochemistry (IHC)-positive isolated tumor cells in the lymph node (N0 \[i+\]) are not considered node-positive and these patients also must have \>= 1 cm residual invasive cancer in the breast to be eligible.
* Patients must not have metastatic disease (i.e., must be clinically M0 or Mx; systemic staging studies with imaging should follow routine practice as per National Comprehensive Cancer Network \[NCCN\] and ASCO guidelines); patients must not have locally recurrent disease
* It is preferred that axillary lymph node sampling is performed after completion of neoadjuvant chemotherapy to allow more accurate assessment of pathologic response; complete axillary lymph node dissection (ALND) after neoadjuvant chemotherapy is recommended in the following situations:
* Patients had documented pathologic involvement of the axillary nodes (fine needle aspiration \[FNA\] or core biopsy) before neoadjuvant chemotherapy and had sentinel node biopsy after neoadjuvant chemotherapy with positive sentinel node(s)
* Patient had documented pathologic involvement of the axillary nodes (FNA or core biopsy) before neoadjuvant chemotherapy and had only 1 sentinel lymph node removed after neoadjuvant chemotherapy
* Except for (i) patients participating in the Alliance A11202 trial) or (ii) if the treating surgeon determines that a complete ALND is not in the best interest of the patient
* NOTE: Patients who undergo sentinel node biopsy before starting neoadjuvant treatment and do not undergo post neoadjuvant assessment of the axillary nodes or who have negative axillary nodes on post neoadjuvant assessment must have \>= 1 cm residual invasive cancer in the breast after completion of neoadjuvant chemotherapy
* Patients must have a minimum of five, available unstained formalin-fixed paraffin-embedded (FFPE) slides (4-5 micron thickness) from the residual (post-neoadjuvant) invasive tumor in primary site or lymph node; (these will be submitted to a central laboratory to determine PD-L1 expression); the tumor tissue must be adequate for PD-L1 testing, which typically requires a minimum of 100 cancer cells per slide; local PD-L1 results, even if available, will not substitute for central testing
* NOTE: Initial order for specimen kits should be placed at least two weeks prior to registering the first patient at each site
* Patients must be offered the opportunity to participate in specimen banking
* The BAHO substudy was permanently closed to accrual on June 23, 2020, and willingness of English-speaking patients to participate in the BAHO substudy is no longer applicable
* Patients must have had neoadjuvant chemotherapy followed by surgery; the choice of neoadjuvant chemotherapy is determined by the treating physician; we recommend following the National Comprehensive Cancer Network (NCCN) treatment guidelines for TNBC; patients who cannot complete all planned treatment cycles for any reason are considered high risk and therefore are eligible for the study if they have residual disease; patients must have resolution of adverse event(s) of the most recent prior chemotherapy to grade 1 or less, except alopecia and =\< grade 2 neuropathy which are allowed
* Patients may receive post-operative (adjuvant) chemotherapy for up to 24 weeks of duration (e.g. 8 cycles of capecitabine as in the Capecitabine for Residual Cancer as Adjuvant Therapy (CREATE-X) trial; the 24-week duration does not include treatment delays) after completion of surgery at the discretion of the treating physician; co-enrollment to EA1131 is allowed, provided that patients complete or discontinue adjuvant chemotherapy prior to step 1 registration; at the time of step 1 registration, patients must have resolution of adverse event(s) of the most recent prior chemotherapy to =\< grade 1, except alopecia and =\< grade 2 neuropathy which are allowed; patients that have received adjuvant chemotherapy (including via co-enrollment to EA1131) must be registered to step 1 within 35 days after final dose of adjuvant chemotherapy
* Patients must have completed their final breast surgery (rendering them free from disease) with clear resection margins for invasive cancer and DCIS within the following timelines:
* 90 days prior to step 1 screening registration for patients not receiving post-operative (adjuvant) chemotherapy OR
* 270 days prior to step 1 screening registration for patients who have received post-operative (adjuvant) chemotherapy Positive margins are allowed only if the surgical team of the patient deems further resection impossible
* Patients for whom radiation therapy (RT) to the affected breast or chest wall and regional nodal areas is clinically indicated as per NCCN treatment guidelines, should receive routine RT after randomization when possible, and receive MK-3475 (pembrolizumab) concurrent with RT, if randomized to the experimental arm; however, routine RT administered, or initiated, prior to registration is also allowed; MK-3475 (pembrolizumab) may be added to ongoing radiation, or started after its completion, if randomized to the experimental arm, provided there are no \> grade 1 radiation-related skin toxicities and provided that no radiosensitizing chemotherapy is being administered; co-enrollment in the Alliance A221505 (NCT03414970) and A011202 (NCT01901094) trials or in the NSABP-B51 (NCT01872975) trial is allowed, but patients must not be planning to receive radiation therapy given on these trials concurrently with MK-3475 (pembrolizumab) treatment on S1418/BR006; whether or not patient will receive RT and the extent of intended RT must be specified at time of registration; NOTE: Patients who receive post-operative chemotherapy may receive radiation therapy before or after the chemotherapy; a short course of reduced dose chemotherapy or other agents concomitant with radiation for radiation sensitization is not considered to be adjuvant chemotherapy
* Patients must not have had prior immunotherapy with anti-PD-L1, anti-PD-1, anti-CTLA4 or similar drugs; patients must not be planning to receive any of the prohibited therapies during the screening or treatment phases of the study
* Patients must not be planning to receive concomitantly other biologic therapy, hormonal therapy, other chemotherapy, surgery or other anti-cancer therapy except radiation therapy while receiving treatment on this protocol; however, patients receiving extended adjuvant endocrine therapy for an earlier ER positive breast cancer treated with curative intent and without recurrence for at least 5 years may continue with their endocrine therapy; elective surgery or surgery that is not related to cancer therapy is allowed, at the discretion of the treating investigator
* Patients must be \>= 18 years of age
* Patients must have Zubrod performance status =\< 2
* Patients must not have a history of (non-infectious) pneumonitis that required steroids or evidence of active pneumonitis within 2 years prior to registration
* Patients must not have active autoimmune disease that has required systemic treatment in past 2 years (i.e., with use of disease modifying agents, corticosteroids or immunosuppressive drugs); replacement therapy (e.g., thyroxine for pre-existing hypothyroidism, insulin for type I diabetes mellitus, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment
* Patients must not have received live vaccines within 30 days prior to registration; examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, chicken pox, shingles, yellow fever, rabies, Bacillus Calmette-Guerin (BCG), and typhoid (oral) vaccine; seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (e.g., Flu-Mist) are live attenuated vaccines, and are not allowed
* Patients must not have known active hepatitis B virus (HBV) or hepatitis C virus (HCV) infection prior to registration; patients who have completed curative therapy for HCV are eligible; patients with known human immunodeficiency virus (HIV) infection are eligible if they meet each of the following 3 criteria:
* CD4 counts \>= 350 mm\^3
* Serum HIV viral load of \< 25,000 IU/ml and
* Treated on a stable antiretroviral regimen
* No other prior invasive malignancy is allowed except for the following: adequately treated basal (or squamous cell) skin cancer, in situ breast or cervical cancer; stage I or II invasive cancer treated with a curative intent without evidence of disease recurrence for at least five years
* Patients must have complete history and physical examination within 28 days prior to registration
* Patients must be informed of the investigational nature of this study and must sign and give written informed consent for this protocol in accordance with institutional and federal guidelines
* As a part of the Oncology Patient Enrollment Network (OPEN) registration process the treating institution's identity is provided in order to ensure that the current (within 365 days) date of institutional review board approval for this study has been entered in the system
* STEP 2 REGISTRATION
* Patients must not be registered to step 2 until receiving confirmation from the Southwest Oncology Group (SWOG) Statistical and Data Management Center that the patient's tissue specimen was adequate for PD-L1 testing; patients must be registered within 14 days of receiving the e-mail notification confirming submission was evaluable for PD-L1 status
* Absolute neutrophil count (ANC) \>= 1,500 microliter (mcL) (obtained within 28 days prior to step 2 registration)
* Platelets \>= 100,000/mcL (obtained within 28 days prior to step 2 registration)
* Hemoglobin \>= 9 g/dL (obtained within 28 days prior to step 2 registration)
* A serum thyroid-stimulating hormone (TSH) and/or free T4 must be obtained within 28 days prior to step 2 registration to obtain a baseline value
* Total bilirubin =\< 1.5 x institutional upper limit of normal (IULN) (except Gilbert's syndrome, who must have a total bilirubin \< 3.0 mg/dL) (obtained within 28 days prior to step 2 registration)
* Serum glutamic-oxaloacetic transaminase (SGOT) (aspartate aminotransferase \[AST\]) or serum glutamate pyruvate transaminase (SGPT) (alanine aminotransferase \[ALT\]) =\< 2.5 x IULN (obtained within 28 days prior to step 2 registration)
* Alkaline phosphatase =\< 2.5 x IULN (obtained within 28 days prior to step 2 registration)
* Serum creatinine =\< IULN OR measured or calculated creatinine clearance \>= 60 mL/min (obtained within 28 days prior to step 2 registration)
* Women of childbearing potential must have a negative urine or serum pregnancy test within 28 day prior to registration; women/men of reproductive potential must have agreed to use an effective contraceptive method for the course of the study through 120 days after the last dose of study medication; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately; a woman is considered to be of "reproductive potential" if she has had menses at any time in the preceding 12 consecutive months; in addition to routine contraceptive methods, "effective contraception" also includes heterosexual celibacy and surgery intended to prevent pregnancy (or with a side-effect of pregnancy prevention) defined as a hysterectomy, bilateral oophorectomy, bilateral tubal ligation, or vasectomy; however, if at any point a previously celibate patient chooses to become heterosexually active during the time period for use of contraceptive measures outlined in the protocol, he/she is responsible for beginning contraceptive measures; patients must not be pregnant or nursing; women of childbearing potential must plan to have a urine or serum pregnancy test within 72 hours prior to receiving the first dose of study medication; if the urine test is positive or cannot be confirmed as negative, a negative serum pregnancy test will be required
* Patients must not have an active infection requiring systemic therapy at the time of starting therapy
* Site must verify that there is no known change in the step 1 eligibility since initial registration
