Inclusion Criteria:

• Patients must have had histologic verification of malignancy at original diagnosis or at relapse, to include any of the following malignancies (no other histology is eligible):
  • Neuroblastoma- measurable
  • Neuroblastoma- MIBG evaluable
  • Rhabdomyosarcoma
  • Osteosarcoma
  • Ewing sarcoma/Peripheral PNET
  • Non-RMS soft tissue sarcoma
  • Hepatoblastoma
  • Malignant germ cell tumor
  • Wilms tumor
  • Acute lymphoblastic leukemia
  • Acute myelogenous leukemia
  • Rhabdoid malignancy
• Disease status for solid tumor patients:
  • Patients must have radiographically measurable disease (with the exception of neuroblastoma)
  • Measurable disease is defined as the presence of at least one lesion on magnetic resonance imaging (MRI) or computed tomography (CT) scan that can be accurately measured with the longest diameter a minimum of 20 mm in at least one dimension; for spiral CT, measurable disease is defined as a minimum diameter of 10 mm in at least one dimension
  • Note: The following do not qualify as measurable disease:
    • Malignant fluid collections (e.g., ascites, pleural effusions)
    • Bone marrow infiltration
    • Lesions detected by nuclear medicine studies (e.g., bone, gallium or positron emission tomography [PET] scans)
    • Elevated tumor markers in plasma or cerebrospinal fluid (CSF)
    • Previously irradiated lesions that have not demonstrated clear progression post radiation
  • Patients with neuroblastoma who do not have measurable disease but have MIBG+ evaluable disease are eligible
• Disease status for leukemia patients:
  • Patients with leukemia must be recurrent or refractory to at least two prior induction or treatment regimens, in addition to the following criteria:
  • Acute lymphoid leukemia:
    • 25% blasts in the bone marrow (M3 bone marrow), excluding patients with known central nervous system (CNS) disease
  • Acute myeloid leukemia according to FAB classification
    • ≥ 5 % blasts in the bone marrow (M2/M3 bone marrow); excluding patients with known CNS disease
• Rhabdoid tumors:
  • To be eligible for enrollment in the rhabdoid tumors stratum, the patient must have a solid tumor where the institutional pathological evaluation of the tumor at initial diagnosis or relapse has confirmed:
    • Morphology and immunophenotypic panel consistent with rhabdoid tumor (required)
    • Loss of SWI/SNF related, matrix associated, actin dependent regulator of chromatin, subfamily b, member 1 (INI1) confirmed by immunohistochemistry, or
    • Molecular confirmation of tumor-specific bi-allelic INI1 loss/mutation if INI1 immunohistochemistry is not available; note that molecular confirmation of tumor-specific bi-allelic INI1 loss/mutation is encouraged in cases where INI1 immunohistochemistry is equivocal
• Patients must have a Lansky or Karnofsky performance status score of ≥ 50, corresponding to Eastern Cooperative Oncology Group (ECOG) categories 0, 1 or 2; use Karnofsky for patients > 16 years of age and Lansky for patients ≤ 16 years of age; Note: Patients who are unable to walk because of paralysis, but who are up in a wheelchair, will be considered ambulatory for the purpose of assessing the performance score
• Patients must have fully recovered from the acute toxic effects of all prior chemotherapy, immunotherapy, or radiotherapy prior to study enrollment
• Myelosuppressive chemotherapy:
  • Solid tumors:
    • Patients with solid tumors must not have received myelosuppressive chemotherapy within 3 weeks of enrollment onto this study (6 weeks if prior nitrosourea)
  • Leukemia:
    • Patients with leukemia who relapse while receiving standard maintenance therapy will not be required to have a waiting period before enrollment onto this study
    • Patients who relapse while they are not receiving standard maintenance therapy must have completely recovered from all acute toxic effects of chemotherapy, immunotherapy or radiotherapy prior to study enrollment; at least 14 days must have elapsed since the completion of cytotoxic therapy, with the exception of hydroxyurea
    • Note: cytoreduction with hydroxyurea can be initiated and continued for up to 24 hours prior to the start of MLN8237
• At least 7 days must have elapsed since the completion of therapy with a growth factor; at least 14 days must have elapsed after receiving pegfilgrastim
• At least 7 days must have elapsed since completion of therapy with a biologic agent; for agents that have known adverse events occurring beyond 7 days after administration, this period prior to enrollment must be extended beyond the time during which adverse events are known to occur
• At least 3 half-lives must have elapsed since prior therapy that included a monoclonal antibody
• ≥ 2 weeks must have elapsed since local palliative radiation therapy (XRT) (small port); ≥ 6 weeks must have elapsed since treatment with therapeutic doses of MIBG; ≥ 6 months must have elapsed if prior craniospinal XRT was received, if ≥ 50% of the pelvis was irradiated, or if total body irradiation (TBI) was received; ≥ 6 weeks must have elapsed if other substantial bone marrow irradiation was given
• No evidence of active graft vs. host disease and ≥ 3 months must have elapsed since transplant
• For patients with solid tumors without bone marrow involvement:
  • Peripheral absolute neutrophil count (ANC) >= 1000/mm^3
  • Platelet count ≥ 100,000/mm^3 (transfusion independent, defined as not receiving platelet transfusions within a 7 day period prior to enrollment)
  • Hemoglobin > 8.0 g/dL (may receive red blood cell [RBC] transfusions)
• For patients with solid tumors and known bone marrow metastatic disease:
  • Peripheral absolute neutrophil count (ANC) ≥ 750/mm^3
  • Platelet count ≥ 50,000/mm^3
  • Hemoglobin ≥ 8.0 g/dL
  • Transfusions are permitted to meet both the platelet and hemoglobin criteria; patients must not be known to be refractory to red blood cell or platelet transfusions
• Patients with leukemia must not be known to be refractory to red blood cell or platelet transfusions
• Creatinine clearance or radioisotope glomerular filtration rate (GFR) 70 mL/min/1.73 m^2 or a serum creatinine based on age/gender as follows:
  • 1 to < 2 years: 0.6
  • 2 to < 6 years: 0.8
  • 6 to < 10 years: 1
  • 10 to < 13 years: 1.2
  • 13 to < 16 years: 1.5 (male), 1.4 (female)
  • >= 16 years: 1.7 (male), 1.4 (female)
• Total bilirubin ≤ 1.5 x upper limit of normal (ULN) for age
• Serum glutamic pyruvate transaminase (SGPT) (alanine aminotransferase [ALT]) ≤ 5.0 x ULN for age (≤ 225 U/L); for the purpose of this study, the ULN for SGPT is 45 U/L
• Serum albumin ≥ 2 g/dL
• All patients and/or their parents or legal guardians must sign a written informed consent

Exclusion Criteria:

• Patients who are pregnant or breast-feeding are not eligible for this study; negative pregnancy tests must be obtained in girls who are post-menarchal; males or females of reproductive potential may not participate unless they have agreed to use an effective contraceptive method for the duration of study therapy; breastfeeding women are excluded
• Growth factors that support platelet or white cell number or function must not have been administered within the 7 days prior to enrollment (14 days if pegfilgrastim)
• Patients requiring corticosteroids who have not been on a stable or decreasing dose of corticosteroid for the 7 days prior to enrollment are not eligible
• Patients who are currently receiving another investigational drug are not eligible
• Patients who are currently receiving other anti-cancer agents are not eligible
• Use of daily benzodiazepine therapy excludes a patient from being eligible because of the potential benzodiazepine-like effects of MLN8237
• Patients who are currently receiving digoxin, cyclosporine, tacrolimus, or sirolimus are not eligible
• Patients who are unable to swallow tablets are not eligible
• Patients who have an uncontrolled infection are not eligible
• Leukemia patients with CNS disease are not eligible
• Patients who in the opinion of the investigator may not be able to comply with the safety monitoring requirements of the study are not eligible