Inclusion Criteria:

• Patients must have a histopathologically confirmed diagnosis of an aggressive B-cell non-Hodgkin lymphoma that is recurrent or refractory to standard therapy.
• For the purpose of this study, aggressive B-cell NHL will be deemed any lymphoma belonging to one of the following groups according to the 2016 revision of the World Health Organization (WHO) classification of lymphoid neoplasms (Swerdlow et al., 2016).
• For the purposes of stratification, diagnoses are grouped into 2 categories:
• Category A
  • Burkitt lymphoma;
  • Burkitt-like lymphoma with 11q aberration;
  • High-grade B-cell lymphoma, with MYC and BCL2 and/or BCL6 rearrangements;
  • High-grade B-cell lymphoma, NOS.
• Category B
  • Diffuse large B-cell lymphoma (DLBCL), NOS;
  • Diffuse large B-cell lymphoma (DLBCL), NOS; Germinal center B-cell type;
  • Diffuse large B-cell lymphoma (DLBCL), NOS; Activated B-cell type;
  • Large B-cell lymphoma with IRF4 rearrangement;
  • T-cell/histiocyte-rich large B-cell lymphoma;
  • Primary DLBCL of the central nervous system (CNS);
  • Primary cutaneous DLBCL, leg type;
  • EBV+ DLBCL, NOS;
  • EBV+ mucocutaneous ulcer;
  • DLBCL associated with chronic inflammation;
  • Lymphomatoid granulomatosis;
  • Primary mediastinal (thymic) large B-cell lymphoma;
  • Intravascular large B-cell lymphoma;
  • ALK+ large B-cell lymphoma ;
  • Plasmablastic lymphoma;
  • Primary effusion lymphoma;
  • HHV-8+ DLBCL, NOS;
  • B-cell lymphoma, unclassifiable, with features intermediate between DLBCL and classical Hodgkin lymphoma. 
• Patients must have measurable disease, defined as at least one lesion that is > 15 mm (1.5 cm) in the longest axis on cross-sectional imaging and measureable in two perpendicular dimensions per computed tomography (spiral CT), PET-CT or MRI.
• Patients must have disease that has relapsed after or is refractory to at least 2 lines of standard therapy; the remaining standard treatment options are unlikely to be effective in the opinion of the treating physician, or patient is felt to be ineligible for such therapies or the patient refuses such therapies; patients who have undergone autologous stem cell transplant are eligible as long as they meet all other criteria.
• Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1. 
• Life expectancy of greater than 12 weeks
• White blood cell (WBC) ≥ 2000/mm^3.
• Absolute neutrophil count (ANC) ≥ 1500/mm^3.
• Platelet count ≥ 100,000/mm^3.
• Hemoglobin > 9.0 g/dL.
• Total bilirubin ≤ 1.5 x upper limit of normal (ULN) (except patients with Gilbert Syndrome, who can have total bilirubin < 3.0 mg/dL). 
• Aspartate transaminase (aspartate aminotransferase [AST]) ≤ 2.5 x ULN. 
• Calculated creatinine clearance (CrCl) ≥ 50 mL/min (if using the Cockcroft-Gault formula). 
• Negative urine or serum pregnancy test result for females of child bearing potential; females of child-bearing potential must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of study participation and for 23 weeks after the last dose of study drug; males who are the sexual partners of a female of child-bearing potential must use any contraceptive method with a failure rate of less than 1% per year for the duration of study participation and for a period of 31 weeks after the last dose of study drug; these periods of required use of contraception have been calculated using the upper limit of the half-life for nivolumab (25 days) and are based on the protocol requirement that females of child-bearing potential use contraception for 5 half-lives plus 30 days and males who are the sexual partners of females of child-bearing potential use contraception for 5 half-lives plus 90 days.
• Females must not be breast-feeding. 
• Females of child bearing potential must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of HCG) within 24 hours prior to the start of nivolumab.
• A female of child-bearing potential is defined as any female who has experienced menarche and who has not undergone surgical sterilization (hysterectomy or bilateral oophorectomy) or who is not postmenopausal.; menopause is defined clinically as 12 months of amenorrhea in a female over 45 in the absence of other biological or physiological causes; in addition, females under the age of 55 must have a documented serum follicle stimulating hormone (FSH) level less than 40 mIU/mL.
• Females who are not of childbearing potential (i.e., who are postmenopausal or surgically sterile) and azoospermic males do not require contraception. 
• Should a female of child-bearing potential become pregnant or suspect she is pregnant while she or her partner is participating in this study, she (or the participating partner) should inform the treating physician immediately.
• Ability to understand and the willingness to sign a written informed consent document.

Exclusion Criteria: 

• Patients who have had chemotherapy or radiotherapy within 4 weeks (6 weeks for nitrosoureas or mitomycin C) or at least 5 half-lives, whichever is longer, prior to entering the study. 
• Palliative (limited-field) radiation therapy is permitted, if all of the following criteria are met. 
• Repeat imaging demonstrates no new sites of bone metastases. 
• The lesion being considered for palliative radiation is not a target lesion. 
• Patients who have not recovered to grade 1 or less from any adverse events due to agents administered more than 4 weeks earlier (excluding alopecia).
• Patients who are receiving any other investigational agents. 
• * Patients should be excluded if they have had prior treatment with an anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CTLA-4 antibody, or any other antibody or drug specifically targeting T-cell co-stimulation or immune checkpoint pathways. 
• Patients who have received autologous stem cell transplant (ASCT) =< 12 weeks prior to the first dose of study drug. 
• Patients with a prior history of allogeneic stem cell or solid organ transplantation. 
• Patients with evidence of active disease in the central nervous system (CNS) defined as either the presence of active lesions on MRI obtained within 4 weeks of registration or progressive neurological decline. 
• Patients with primary CNS lymphoma who develop systemic recurrence following standard therapy may be included as long as no active CNS disease is present at the time or enrollment; similarly, patients with secondary involvement of the CNS from a systemic lymphoma may be included as long as the CNS disease has been optimally treated and they demonstrate no evidence of active CNS disease. 
• History of allergic reactions attributed to compounds of similar chemical or biologic composition to CDX-1127 (varlilumab) and/or nivolumab.
• History of severe hypersensitivity reaction to any monoclonal antibody. 
• Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements. 
• Pregnant women are excluded from this study; breastfeeding should be discontinued if the mother is treated with CDX-1127 (varlilumab) or nivolumab .
• Patients with human immunodeficiency virus (HIV) are eligible for the study provided they meet the other protocol criteria in addition to the following: 
  • Undetectable HIV load by standard polymerase chain reaction (PCR) clinical assay; 
  • Absolute CD4 count of >= 200 mm^3;
  • Willing to maintain adherence to combination antiretroviral therapy; 
  • No history of acquired immunodeficiency syndrome (AIDS) defining condition (other than lymphoma or CD4 cell count < 200 mm^3); 
  • Likely to have near normal lifespan if not for the presence of relapsed/refractory lymphoma:
    • Patients with evidence of hepatitis B virus (HBV) are eligible provided there is minimal hepatic injury and the patient has undetectable HBV on suppressive HBV therapy; patient must be willing to maintain adherence to HBV therapy; 
    • Patients with previously treated and eradicated hepatitis C virus (HCV) who have minimal hepatic injury are eligible. 
• Patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including systemic corticosteroids, should be excluded; these include but are not limited to patients with a history of immune related neurologic disease, multiple sclerosis, autoimmune (demyelinating) neuropathy, Guillain-Barre syndrome, myasthenia gravis; systemic autoimmune disease such as systemic lupus erythematosus (SLE), connective tissue diseases, scleroderma, inflammatory bowel disease (IBD), Crohn's, ulcerative colitis, hepatitis; and patients with a history of toxic epidermal necrolysis (TEN), Stevens-Johnson syndrome, or phospholipid syndrome should be excluded because of the risk of recurrence or exacerbation of disease; patients with vitiligo, endocrine deficiencies including thyroiditis managed with replacement hormones including physiologic corticosteroids are eligible; patients with rheumatoid arthritis and other arthropathies, Sjogren's syndrome and psoriasis controlled with topical medication and patients with positive serology, such as antinuclear antibodies (ANA), anti-thyroid antibodies should be evaluated for the presence of target organ involvement and potential need for systemic treatment but should otherwise be eligible. 
• Patients are permitted to enroll if they have vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger (precipitating event). 
• Patients should be excluded if they have a condition requiring systemic treatment with either corticosteroids (> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study drug administration; inhaled or topical steroids and adrenal replacement doses =< 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease; patients are permitted to use topical, ocular, intra-articular, intranasal, and inhalational corticosteroids (with minimal systemic absorption); physiologic replacement doses of systemic corticosteroids are permitted, even if =< 10 mg/day prednisone equivalents; a brief course of corticosteroids for prophylaxis (e.g., contrast dye allergy) or for treatment of non-autoimmune conditions (e.g., delayed-type hypersensitivity reaction caused by contact allergen) is permitted. 
• Patients who have had evidence of active or acute diverticulitis, intra-abdominal abscess, gastrointestinal (GI) obstruction and abdominal carcinomatosis which are known risk factors for bowel perforation should be evaluated for the potential need for additional treatment before coming on study. 
• Patients with other active malignancy =< 3 years prior to registration for which active treatment is required must be excluded; patients with composite lymphomas that have a non-B-cell component must be excluded.
  • EXCEPTIONS: Non-melanotic skin cancer or carcinoma-in-situ of the cervix.