Inclusion Criteria:
* Parts 1, 2, and 5: Participants with histologically or cytologically confirmed metastatic castration-resistant prostate cancer (mCRPC) who are refractory to a novel antiandrogen therapy (abiraterone acetate and/or enzalutamide, apalutamide, or darolutamide) and have failed at least 1 (but not more than 2) taxane regimens including for metastatic hormone-sensitive prostate cancer (mHSPC) (or who are deemed medically unsuitable to be treated with a taxane regimen or have actively refused treatment with a taxane regimen). Note: A taxane regimen is defined as a minimum exposure of 2 cycles of a taxane. Any NHT that has been administered and has been stopped for reasons other than progression will not be counted as an additional line of treatment.
1. Dose exploration phase: Novel antiandrogen therapy must have been given for treatment of metastatic disease.
2. Dose expansion phase: participants must not have had more than 2 NHTs and 2 taxane regimens in any setting, and an additional up to 2 other systemic anti-cancer treatments are allowed (eg, anti-PD1, PARP inhibitors, radioligand therapies, sipuleucel-T, experimental agents) Note: Combinations are considered one systemic anti-cancer treatment.
* Part 3: Participants with histologically or cytologically confirmed mCRPC who are refractory to a NHT (abiraterone acetate, enzalutamide, apalutamide, or darolutamide) given in any disease setting and who are deemed medically unsuitable to be treated with a taxane regimen or have actively refused treatment with a taxane regimen (no prior taxanes). 0-1 prior PARP inhibitors or sipuleucel-T treatments are acceptable. Participants who received prior investigational therapy for the treatment of metastatic disease are not eligible.
* Parts 4A and 4B:
1. Participants with histologically or cytologically confirmed mCRPC who are refractory to 0-2 prior NHT (given in any disease setting depending on the part), and no or 1 taxane (given for hormone sensitive prostate cancer).
2. Dose-expansion phase: at least 1 prior NHT must have been given; 0-1 prior PARP inhibitors are acceptable.
3. 4A: Participants planning to receive abiraterone acetate for the first time (participants who received prior abiraterone acetate are not eligible). Participants may have had exposure to up to 2 NHTs with a similar mechanism of action (apalutamide, enzalutamide or darolutamide) in the non-mCRPC and mCRPC setting.
4. 4B: Participants planning to receive enzalutamide for the first time (participants who received prior enzalutamide/apalutamide or daralutamide are not eligible).
* All parts:
* Participants must have undergone bilateral orchiectomy or be on continuous androgen-deprivation therapy with a gonadotropin releasing hormone (GnRH) agonist or antagonist.
* Total serum testosterone \<= 50 ng/dL or 1.7 nmol/L.
* Evidence of progressive disease, defined as 1 or more Prostate Cancer Working Group 3 (PCWG3) criteria:
1. PSA level \>= 1 ng/mL that has increased on at least 2 successive occasions at least 1 week apart.
2. nodal or visceral progression as defined by Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 with PCGW3 modifications.
3. appearance of 2 or more new lesions in bone scan.
* Eastern Cooperative Oncology Group performance status of 0-1.
* Adequate organ function, defined as follows:
1. Hematological function:
1. absolute neutrophil count \>= 1 x 10\^9/L (without growth factor support within 7 days from screening assessment).
2. platelet count \>= 75 x 10\^9/L (without platelet transfusion within 7 days from screening assessment).
3. hemoglobin \>= 9 g/dL (90 g/L) (without blood transfusion within 7 days from screening assessment).
2. Renal function:
1. estimated glomerular filtration rate based on Modification of Diet in Renal Disease calculation \>= 30 ml/min/1.73 m\^2.
3. Hepatic function:
1. aspartate aminotransferase (AST) and alanine aminotransferase (ALT) \< 3 x upper limit of normal (ULN) (or \< 5 x ULN for participants with liver involvement).
2. total bilirubin (TBL) \< 1.5 x ULN (or \< 2 x ULN for participants with liver metastases).
4. Cardiac function:
1. left ventricular ejection fraction \> 50% (2-D transthoracic echocardiogram \[ECHO\] is the preferred method of evaluation; multi-gated acquisition scan is acceptable if ECHO is not available).
2. Baseline electrocardiogram (ECG) QTcF \<= 470 msec (average of triplicate values).
Part 3-Retreatment group:
* Deriving benefit from initial treatment with AMG 509 as evidenced by one of the following:
1. confirmed PSA50 response.
2. radiographic stable disease/partial response/complete response during 6 cycles of initial treatment with AMG 509 and without progression during the first 6 cycles.
* No discontinuation for toxicity during the initial treatment with 6 cycles of AMG 509.
* Progressive disease as defined in I106 within 12 months of final dose in their initial treatment with 6 cycles (EOT_1).
* Willingness to have a fresh tumor biopsy prior to initiating the additional course of treatment, depending on safety and feasibility as assessed by investigator.
Exclusion Criteria:
* Pathological finding consistent with pure small cell, neuroendocrine carcinoma of the prostate or any other histology different from adenocarcinoma.
* Radiation therapy within 4 weeks of first dose (or local or focal radiotherapy within 2 weeks of first dose).
* Untreated central nervous system (CNS) metastases or leptomeningeal disease. Participants with a history of treated CNS metastases are eligible if there is radiographic evidence of improvement upon the completion of CNS-directed therapy and no evidence of interim progression between the completion of CNS-directed therapy and the screening radiographic study.
* Participants with symptoms and/or clinical signs and/or radiographic signs that indicate an acute and/or uncontrolled active systemic infection within 7 days prior to the first dose of investigational product administration.
* Confirmed history or current autoimmune disease or other diseases resulting in permanent immunosuppression or requiring permanent immunosuppressive therapy.
* History of arterial or venous thrombosis (eg, stroke, transient ischemic attack, pulmonary embolism, or deep vein thrombosis); for arterial thrombosis within 12 months of AMG 509 initiation; for venous thrombosis, 6 months and stable on anti-coagulation.
* Myocardial infarction and/or symptomatic congestive heart failure (New York Heart Association \> class II) within 12 months of first dose of AMG 509 with the exception of ischemia or non-ST segment elevation myocardial infarction controlled with stent placement and confirmed by a cardiologist more than 6 months prior to first dose of AMG 509.
* Any anti-cancer therapy or immunotherapy within 4 weeks of start of first dose, not including luteinizing hormone-releasing hormone (LHRH)/GnRH analogue (agonist/antagonist).
* Prior PSMA radionuclide therapy within 2 months prior to AMG 509 unless participant received \< 2 cycles (Note: a participant cannot have received PSMA radionuclide therapy \< 35 days prior to enrollment if 1 cycle was given). Parts 3 and 4: prior PSMA radionuclide therapy is prohibited. Participants on a stable bisphosphonate or denosumab regimen for \>= 30 days prior to enrollment are eligible (exception: part 3 retreatment).
* Part 3-Retreatment only: Any anticancer therapy or immunotherapy, not including luteinizing hormone-releasing hormone/gonadotropin releasing hormone (LHRH/GnRH) analogue (agonist/antagonist), and/or bisphosphonate or denosumab regimen after last dose of AMG 509 initial course of treatment.
