Skip to content
Projects
Groups
Snippets
Help
Loading...
Sign in
Toggle navigation
C
cromwell-sdk
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
liugaoling
cromwell-sdk
Commits
a4e75954
Commit
a4e75954
authored
May 11, 2022
by
liugaoling
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Initialized
parent
1d48d174
Pipeline
#151
canceled with stages
Changes
23
Pipelines
1
Expand all
Hide whitespace changes
Inline
Side-by-side
Showing
23 changed files
with
980 additions
and
0 deletions
+980
-0
.gitignore
.gitignore
+34
-0
mvnw
mvnw
+0
-0
mvnw.cmd
mvnw.cmd
+188
-0
pom.xml
pom.xml
+67
-0
MANIFEST.MF
src/main/java/META-INF/MANIFEST.MF
+3
-0
CloudamClient.java
src/main/java/com/cloudam/cromwell/CloudamClient.java
+72
-0
CloudamRequest.java
src/main/java/com/cloudam/cromwell/CloudamRequest.java
+12
-0
CloudamResponse.java
src/main/java/com/cloudam/cromwell/CloudamResponse.java
+16
-0
CloudamURL.java
src/main/java/com/cloudam/cromwell/constant/CloudamURL.java
+21
-0
CreateJobRequest.java
...main/java/com/cloudam/cromwell/core/CreateJobRequest.java
+92
-0
CreateJobResponse.java
...ain/java/com/cloudam/cromwell/core/CreateJobResponse.java
+7
-0
GetJobRequest.java
src/main/java/com/cloudam/cromwell/core/GetJobRequest.java
+42
-0
GetJobResponse.java
src/main/java/com/cloudam/cromwell/core/GetJobResponse.java
+8
-0
WorkSpaceFactory.java
...main/java/com/cloudam/cromwell/core/WorkSpaceFactory.java
+21
-0
AutoCluster.java
src/main/java/com/cloudam/cromwell/domain/AutoCluster.java
+54
-0
CLoudamDisk.java
src/main/java/com/cloudam/cromwell/domain/CLoudamDisk.java
+16
-0
Command.java
src/main/java/com/cloudam/cromwell/domain/Command.java
+27
-0
Job.java
src/main/java/com/cloudam/cromwell/domain/Job.java
+76
-0
JobDescription.java
...main/java/com/cloudam/cromwell/domain/JobDescription.java
+128
-0
MountEntry.java
src/main/java/com/cloudam/cromwell/domain/MountEntry.java
+36
-0
Os.java
src/main/java/com/cloudam/cromwell/domain/Os.java
+7
-0
HttpUtil.java
src/main/java/com/cloudam/cromwell/http/HttpUtil.java
+52
-0
application.properties
src/main/resources/application.properties
+1
-0
No files found.
.gitignore
0 → 100644
View file @
a4e75954
HELP.md
target/
!.mvn/wrapper/maven-wrapper.jar
!**/src/main/**/target/
!**/src/test/**/target/
### STS ###
.apt_generated
.classpath
.factorypath
.project
.settings
.springBeans
.sts4-cache
### IntelliJ IDEA ###
.idea
*.iws
*.iml
*.ipr
.mvn
### NetBeans ###
/nbproject/private/
/nbbuild/
/dist/
/nbdist/
/.nb-gradle/
build/
!**/src/main/**/build/
!**/src/test/**/build/
### VS Code ###
.vscode/
mvnw
0 → 100644
View file @
a4e75954
This diff is collapsed.
Click to expand it.
mvnw.cmd
0 → 100644
View file @
a4e75954
@REM ----------------------------------------------------------------------------
@REM Licensed to the Apache Software Foundation (ASF) under one
@REM or more contributor license agreements. See the NOTICE file
@REM distributed with this work for additional information
@REM regarding copyright ownership. The ASF licenses this file
@REM to you under the Apache License, Version 2.0 (the
@REM "License"); you may not use this file except in compliance
@REM with the License. You may obtain a copy of the License at
@REM
@REM https://www.apache.org/licenses/LICENSE-2.0
@REM
@REM Unless required by applicable law or agreed to in writing,
@REM software distributed under the License is distributed on an
@REM "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@REM KIND, either express or implied. See the License for the
@REM specific language governing permissions and limitations
@REM under the License.
@REM ----------------------------------------------------------------------------
@REM ----------------------------------------------------------------------------
@REM Maven Start Up Batch script
@REM
@REM Required ENV vars:
@REM JAVA_HOME - location of a JDK home dir
@REM
@REM Optional ENV vars
@REM M2_HOME - location of maven2's installed home dir
@REM MAVEN_BATCH_ECHO - set to 'on' to enable the echoing of the batch commands
@REM MAVEN_BATCH_PAUSE - set to 'on' to wait for a keystroke before ending
@REM MAVEN_OPTS - parameters passed to the Java VM when running Maven
@REM e.g. to debug Maven itself, use
@REM set MAVEN_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000
@REM MAVEN_SKIP_RC - flag to disable loading of mavenrc files
@REM ----------------------------------------------------------------------------
@REM Begin all REM lines with '@' in case MAVEN_BATCH_ECHO is 'on'
@echo off
@REM set title of command window
title %0
@REM enable echoing by setting MAVEN_BATCH_ECHO to 'on'
@if "%MAVEN_BATCH_ECHO%" == "on" echo %MAVEN_BATCH_ECHO%
@REM set %HOME% to equivalent of $HOME
if "%HOME%" == "" (set "HOME=%HOMEDRIVE%%HOMEPATH%")
@REM Execute a user defined script before this one
if not "%MAVEN_SKIP_RC%" == "" goto skipRcPre
@REM check for pre script, once with legacy .bat ending and once with .cmd ending
if exist "%USERPROFILE%\mavenrc_pre.bat" call "%USERPROFILE%\mavenrc_pre.bat" %*
if exist "%USERPROFILE%\mavenrc_pre.cmd" call "%USERPROFILE%\mavenrc_pre.cmd" %*
:skipRcPre
@setlocal
set ERROR_CODE=0
@REM To isolate internal variables from possible post scripts, we use another setlocal
@setlocal
@REM ==== START VALIDATION ====
if not "%JAVA_HOME%" == "" goto OkJHome
echo.
echo Error: JAVA_HOME not found in your environment. >&2
echo Please set the JAVA_HOME variable in your environment to match the >&2
echo location of your Java installation. >&2
echo.
goto error
:OkJHome
if exist "%JAVA_HOME%\bin\java.exe" goto init
echo.
echo Error: JAVA_HOME is set to an invalid directory. >&2
echo JAVA_HOME = "%JAVA_HOME%" >&2
echo Please set the JAVA_HOME variable in your environment to match the >&2
echo location of your Java installation. >&2
echo.
goto error
@REM ==== END VALIDATION ====
:init
@REM Find the project base dir, i.e. the directory that contains the folder ".mvn".
@REM Fallback to current working directory if not found.
set MAVEN_PROJECTBASEDIR=%MAVEN_BASEDIR%
IF NOT "%MAVEN_PROJECTBASEDIR%"=="" goto endDetectBaseDir
set EXEC_DIR=%CD%
set WDIR=%EXEC_DIR%
:findBaseDir
IF EXIST "%WDIR%"\.mvn goto baseDirFound
cd ..
IF "%WDIR%"=="%CD%" goto baseDirNotFound
set WDIR=%CD%
goto findBaseDir
:baseDirFound
set MAVEN_PROJECTBASEDIR=%WDIR%
cd "%EXEC_DIR%"
goto endDetectBaseDir
:baseDirNotFound
set MAVEN_PROJECTBASEDIR=%EXEC_DIR%
cd "%EXEC_DIR%"
:endDetectBaseDir
IF NOT EXIST "%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config" goto endReadAdditionalConfig
@setlocal EnableExtensions EnableDelayedExpansion
for /F "usebackq delims=" %%a in ("%MAVEN_PROJECTBASEDIR%\.mvn\jvm.config") do set JVM_CONFIG_MAVEN_PROPS=!JVM_CONFIG_MAVEN_PROPS! %%a
@endlocal & set JVM_CONFIG_MAVEN_PROPS=%JVM_CONFIG_MAVEN_PROPS%
:endReadAdditionalConfig
SET MAVEN_JAVA_EXE="%JAVA_HOME%\bin\java.exe"
set WRAPPER_JAR="%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.jar"
set WRAPPER_LAUNCHER=org.apache.maven.wrapper.MavenWrapperMain
set DOWNLOAD_URL="https://repo.maven.apache.org/maven2/org/apache/maven/wrapper/maven-wrapper/3.1.0/maven-wrapper-3.1.0.jar"
FOR /F "usebackq tokens=1,2 delims==" %%A IN ("%MAVEN_PROJECTBASEDIR%\.mvn\wrapper\maven-wrapper.properties") DO (
IF "%%A"=="wrapperUrl" SET DOWNLOAD_URL=%%B
)
@REM Extension to allow automatically downloading the maven-wrapper.jar from Maven-central
@REM This allows using the maven wrapper in projects that prohibit checking in binary data.
if exist %WRAPPER_JAR% (
if "%MVNW_VERBOSE%" == "true" (
echo Found %WRAPPER_JAR%
)
) else (
if not "%MVNW_REPOURL%" == "" (
SET DOWNLOAD_URL="%MVNW_REPOURL%/org/apache/maven/wrapper/maven-wrapper/3.1.0/maven-wrapper-3.1.0.jar"
)
if "%MVNW_VERBOSE%" == "true" (
echo Couldn't find %WRAPPER_JAR%, downloading it ...
echo Downloading from: %DOWNLOAD_URL%
)
powershell -Command "&{"^
"$webclient = new-object System.Net.WebClient;"^
"if (-not ([string]::IsNullOrEmpty('%MVNW_USERNAME%') -and [string]::IsNullOrEmpty('%MVNW_PASSWORD%'))) {"^
"$webclient.Credentials = new-object System.Net.NetworkCredential('%MVNW_USERNAME%', '%MVNW_PASSWORD%');"^
"}"^
"[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; $webclient.DownloadFile('%DOWNLOAD_URL%', '%WRAPPER_JAR%')"^
"}"
if "%MVNW_VERBOSE%" == "true" (
echo Finished downloading %WRAPPER_JAR%
)
)
@REM End of extension
@REM Provide a "standardized" way to retrieve the CLI args that will
@REM work with both Windows and non-Windows executions.
set MAVEN_CMD_LINE_ARGS=%*
%MAVEN_JAVA_EXE% ^
%JVM_CONFIG_MAVEN_PROPS% ^
%MAVEN_OPTS% ^
%MAVEN_DEBUG_OPTS% ^
-classpath %WRAPPER_JAR% ^
"-Dmaven.multiModuleProjectDirectory=%MAVEN_PROJECTBASEDIR%" ^
%WRAPPER_LAUNCHER% %MAVEN_CONFIG% %*
if ERRORLEVEL 1 goto error
goto end
:error
set ERROR_CODE=1
:end
@endlocal & set ERROR_CODE=%ERROR_CODE%
if not "%MAVEN_SKIP_RC%"=="" goto skipRcPost
@REM check for post script, once with legacy .bat ending and once with .cmd ending
if exist "%USERPROFILE%\mavenrc_post.bat" call "%USERPROFILE%\mavenrc_post.bat"
if exist "%USERPROFILE%\mavenrc_post.cmd" call "%USERPROFILE%\mavenrc_post.cmd"
:skipRcPost
@REM pause the script if MAVEN_BATCH_PAUSE is set to 'on'
if "%MAVEN_BATCH_PAUSE%"=="on" pause
if "%MAVEN_TERMINATE_CMD%"=="on" exit %ERROR_CODE%
cmd /C exit /B %ERROR_CODE%
pom.xml
0 → 100644
View file @
a4e75954
<?xml version="1.0" encoding="UTF-8"?>
<project
xmlns=
"http://maven.apache.org/POM/4.0.0"
xmlns:xsi=
"http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=
"http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"
>
<modelVersion>
4.0.0
</modelVersion>
<groupId>
com.cloudam
</groupId>
<artifactId>
cromwell
</artifactId>
<version>
1.0.0-SNAPSHOT
</version>
<name>
cromwell-sdk
</name>
<description>
cloudam support cromwell sdk
</description>
<properties>
<java.version>
1.8
</java.version>
</properties>
<dependencies>
<dependency>
<groupId>
com.alibaba
</groupId>
<artifactId>
fastjson
</artifactId>
<version>
1.2.71
</version>
</dependency>
<dependency>
<groupId>
com.squareup.okhttp3
</groupId>
<artifactId>
okhttp
</artifactId>
<version>
3.8.1
</version>
</dependency>
<dependency>
<groupId>
ch.qos.logback
</groupId>
<artifactId>
logback-classic
</artifactId>
<version>
1.2.10
</version>
</dependency>
<dependency>
<groupId>
commons-lang
</groupId>
<artifactId>
commons-lang
</artifactId>
<version>
2.6
</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>
maven-assembly-plugin
</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>
jar-with-dependencies
</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>
make-assembly
</id>
<phase>
package
</phase>
<goals>
<goal>
single
</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>
org.apache.maven.plugins
</groupId>
<artifactId>
maven-compiler-plugin
</artifactId>
<configuration>
<source>
8
</source>
<target>
8
</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
src/main/java/META-INF/MANIFEST.MF
0 → 100644
View file @
a4e75954
Manifest-Version: 1.0
Main-Class: com.cloudam.cromwell.CloudamClient
src/main/java/com/cloudam/cromwell/CloudamClient.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
;
import
com.alibaba.fastjson.JSON
;
import
com.alibaba.fastjson.JSONObject
;
import
com.cloudam.cromwell.constant.CloudamURL
;
import
com.cloudam.cromwell.core.CreateJobRequest
;
import
com.cloudam.cromwell.core.CreateJobResponse
;
import
com.cloudam.cromwell.core.WorkSpaceFactory
;
import
com.cloudam.cromwell.domain.Job
;
import
com.cloudam.cromwell.domain.JobDescription
;
import
com.cloudam.cromwell.http.HttpUtil
;
import
okhttp3.*
;
import
org.slf4j.Logger
;
import
org.slf4j.LoggerFactory
;
public
class
CloudamClient
{
private
final
String
workSpace
;
private
final
String
accessToken
;
private
final
Logger
logger
=
LoggerFactory
.
getLogger
(
CloudamClient
.
class
);
public
CloudamClient
(
String
env
,
String
accessToken
)
{
this
.
workSpace
=
env
;
this
.
accessToken
=
accessToken
;
auth
();
}
private
void
auth
()
{
String
baseUrl
=
WorkSpaceFactory
.
getBaseUrl
(
this
.
workSpace
);
// 这里是发送请求校验token是否合法
Request
.
Builder
request
=
HttpUtil
.
getBuilder
(
this
.
accessToken
)
.
url
(
baseUrl
+
CloudamURL
.
AUTH_URL_SUFFIX
)
.
get
();
Response
response
=
HttpUtil
.
execNormalHttpRequest
(
request
,
3
);
if
(
response
==
null
||
response
.
code
()
!=
200
||
response
.
body
()
==
null
)
{
throw
new
IllegalArgumentException
(
"the user access token invalid:"
+
this
.
accessToken
);
}
}
public
<
T
extends
CloudamResponse
>
T
doAction
(
CloudamRequest
<
T
>
cloudamRequest
)
{
String
baseUrl
=
WorkSpaceFactory
.
getBaseUrl
(
this
.
workSpace
);
CloudamResponse
cloudamResponse
=
null
;
try
{
Request
.
Builder
request
=
cloudamRequest
.
getBuilder
(
HttpUtil
.
getBuilder
(
this
.
accessToken
),
baseUrl
,
cloudamRequest
);
Response
response
=
HttpUtil
.
execNormalHttpRequest
(
request
,
3
);
if
(
response
!=
null
&&
response
.
code
()
==
200
&&
response
.
body
()
!=
null
)
{
Job
job
=
JSON
.
parseObject
(
response
.
body
().
bytes
(),
Job
.
class
);
cloudamResponse
=
cloudamRequest
.
getResponseClass
().
getDeclaredConstructor
().
newInstance
();
cloudamResponse
.
setJob
(
job
);
}
}
catch
(
Exception
e
)
{
logger
.
error
(
"Failed to invoke http request from cloudam, method: {}, request param: {}, e:{}"
,
cloudamRequest
.
getClass
().
getSimpleName
(),
JSON
.
toJSONString
(
cloudamRequest
),
e
);
}
logger
.
info
(
"class {}, response:{}"
,
cloudamRequest
.
getResponseClass
(),
JSON
.
toJSONString
(
cloudamResponse
));
return
(
T
)
cloudamResponse
;
}
public
static
void
main
(
String
[]
args
)
{
CreateJobRequest
createJobRequest
=
new
CreateJobRequest
();
String
json
=
"{\"autoCluster\":{\"os\":\"Linux\",\"partition\":\"c-4-1\",\"type\":\"OnDemand\"},\"cpus\":1,\"envs\":{\"cwd\":\"D:\\\\cromwell\\\\slurm\\\\a9d00301-1fed-4607-aa79-86d200914050\\\\call-task_A\",\"stdout\":\"D:\\\\cromwell\\\\slurm\\\\a9d00301-1fed-4607-aa79-86d200914050\\\\call-task_A\\\\stdout\",\"stderr\":\"D:\\\\cromwell\\\\slurm\\\\a9d00301-1fed-4607-aa79-86d200914050\\\\call-task_A\\\\stderr\"},\"gpus\":0,\"name\":\"cloudam_a9d00304_task_A\",\"ntasks\":2,\"projectId\":\"aaa\",\"timeout\":10000,\"zoneId\":\"CE-Z1\"}"
;
JobDescription
jobDescription
=
JSONObject
.
parseObject
(
json
,
JobDescription
.
class
);
createJobRequest
.
setJobDescription
(
jobDescription
);
CloudamClient
cloudamClient
=
new
CloudamClient
(
"release"
,
"61f00e1a-3142-405e-8298-b7b2bab41d28"
);
CreateJobResponse
createJobResponse
=
cloudamClient
.
doAction
(
createJobRequest
);
System
.
out
.
println
(
JSON
.
toJSONString
(
createJobResponse
));
}
}
src/main/java/com/cloudam/cromwell/CloudamRequest.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
;
import
okhttp3.Request
;
public
abstract
class
CloudamRequest
<
T
extends
CloudamResponse
>
{
String
urlSuffix
=
""
;
public
abstract
Class
<
T
>
getResponseClass
();
public
abstract
Request
.
Builder
getBuilder
(
Request
.
Builder
builder
,
String
url
,
CloudamRequest
<
T
>
cloudamRequest
);
}
src/main/java/com/cloudam/cromwell/CloudamResponse.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
;
import
com.cloudam.cromwell.domain.Job
;
public
abstract
class
CloudamResponse
{
private
Job
job
;
public
Job
getJob
()
{
return
job
;
}
public
void
setJob
(
Job
job
)
{
this
.
job
=
job
;
}
}
src/main/java/com/cloudam/cromwell/constant/CloudamURL.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
constant
;
public
interface
CloudamURL
{
String
BASE_DEV_URL
=
"http://47.57.4.97/"
;
String
BASE_RELEASE_URL
=
"http://47.75.104.171/"
;
String
BASE_PROD_URL
=
""
;
String
AUTH_URL_SUFFIX
=
"uaa/users/current"
;
String
CREATE_JOB_URL_SUFFIX
=
"c3ce/cloud-bursting/job"
;
String
GET_JOB_URL_SUFFIX
=
"c3ce/cloud-bursting/job/request"
;
String
STOP_JOB_URL_SUFFIX
=
"c3ce/cloud-bursting/job"
;
String
DELETE_JOB_URL_SUFFIX
=
""
;
}
src/main/java/com/cloudam/cromwell/core/CreateJobRequest.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
core
;
import
com.alibaba.fastjson.JSON
;
import
com.alibaba.fastjson.JSONObject
;
import
com.cloudam.cromwell.CloudamRequest
;
import
com.cloudam.cromwell.constant.CloudamURL
;
import
com.cloudam.cromwell.domain.JobDescription
;
import
com.cloudam.cromwell.domain.Os
;
import
okhttp3.MediaType
;
import
okhttp3.Request
;
import
org.apache.commons.lang.StringUtils
;
import
org.slf4j.Logger
;
import
org.slf4j.LoggerFactory
;
import
java.util.Base64
;
import
java.util.function.BiConsumer
;
public
class
CreateJobRequest
extends
CloudamRequest
<
CreateJobResponse
>
{
private
final
String
urlSuffix
=
CloudamURL
.
CREATE_JOB_URL_SUFFIX
;
private
final
Logger
logger
=
LoggerFactory
.
getLogger
(
CreateJobRequest
.
class
);
private
JobDescription
jobDescription
;
public
void
setJobDescription
(
JobDescription
jobDescription
)
{
this
.
jobDescription
=
jobDescription
;
}
public
JobDescription
getJobDescription
()
{
return
jobDescription
;
}
@Override
public
Class
<
CreateJobResponse
>
getResponseClass
()
{
return
CreateJobResponse
.
class
;
}
@Override
public
Request
.
Builder
getBuilder
(
Request
.
Builder
builder
,
String
url
,
CloudamRequest
<
CreateJobResponse
>
cloudamRequest
)
{
CreateJobRequest
createJobRequest
=
(
CreateJobRequest
)
cloudamRequest
;
return
builder
.
url
(
url
+
createJobRequest
.
urlSuffix
)
.
post
(
okhttp3
.
RequestBody
.
create
(
MediaType
.
parse
(
"charset=utf-8"
),
JSON
.
toJSONBytes
(
buildJson
(
createJobRequest
.
getJobDescription
()))));
}
private
JSONObject
buildJson
(
JobDescription
jobDescription
)
{
JSONObject
jsonObject
=
new
JSONObject
();
logger
.
info
(
"createJobRequest JobDescription:{}"
,
JSON
.
toJSONString
(
jobDescription
));
jsonObject
.
put
(
"envs"
,
jobDescription
.
getEnvs
());
jsonObject
.
put
(
"jobName"
,
jobDescription
.
getName
());
jsonObject
.
put
(
"projectId"
,
jobDescription
.
getProjectId
());
jsonObject
.
put
(
"zoneId"
,
jobDescription
.
getZoneId
());
jsonObject
.
put
(
"os"
,
jobDescription
.
getAutoCluster
().
getOs
());
jsonObject
.
put
(
"slurmCommand"
,
Base64
.
getEncoder
().
encodeToString
(
buildSbatchSubmit
(
jobDescription
).
getBytes
()));
jsonObject
.
put
(
"masterId"
,
"6279cc3480533c0001adbf8b"
);
System
.
out
.
println
(
jsonObject
.
toJSONString
());
return
jsonObject
;
}
private
String
buildSbatchSubmit
(
JobDescription
jobDescription
)
{
// sbatch -n 1 -c 8 --comment disk:CLOUD_SSD:2700G:s-NV0tQ0PL run_local.sh
// sbatch -n 1 -c 8 -p c-52-2 -q eco cpu.sh
StringBuffer
commandBuffer
=
new
StringBuffer
(
"#!/bin/sh\n"
);
BiConsumer
<
String
,
String
>
stringValidAppend
=
(
key
,
agrs
)
->
{
if
(
StringUtils
.
isNotBlank
(
key
))
{
commandBuffer
.
append
(
"#SBATCH "
).
append
(
agrs
).
append
(
" "
).
append
(
key
).
append
(
"\n"
);
}
};
BiConsumer
<
Integer
,
String
>
intValidAppend
=
(
key
,
agrs
)
->
{
if
(
key
!=
null
&&
key
!=
0
)
{
commandBuffer
.
append
(
"#SBATCH "
).
append
(
agrs
).
append
(
" "
).
append
(
key
).
append
(
"\n"
);
}
};
stringValidAppend
.
accept
(
jobDescription
.
getName
(),
"-J"
);
stringValidAppend
.
accept
(
jobDescription
.
getAutoCluster
().
getPartition
(),
"-p"
);
intValidAppend
.
accept
(
jobDescription
.
getCpus
(),
"-c"
);
intValidAppend
.
accept
(
jobDescription
.
getGpus
(),
"-g"
);
intValidAppend
.
accept
(
jobDescription
.
getNtasks
(),
"-n"
);
intValidAppend
.
accept
(
jobDescription
.
getTimeout
(),
"-t"
);
String
workScript
=
jobDescription
.
getEnvs
().
get
(
"cwd"
)
+
"/script"
;
if
(
Os
.
Windows
!=
jobDescription
.
getAutoCluster
().
getOs
())
{
commandBuffer
.
append
(
"dos2unix "
).
append
(
workScript
).
append
(
"\n"
);
}
commandBuffer
.
append
(
"bash "
).
append
(
workScript
).
append
(
"\n"
);
return
commandBuffer
.
toString
();
}
}
src/main/java/com/cloudam/cromwell/core/CreateJobResponse.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
core
;
import
com.cloudam.cromwell.CloudamResponse
;
public
class
CreateJobResponse
extends
CloudamResponse
{
}
src/main/java/com/cloudam/cromwell/core/GetJobRequest.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
core
;
import
com.cloudam.cromwell.CloudamRequest
;
import
com.cloudam.cromwell.constant.CloudamURL
;
import
okhttp3.Request
;
public
class
GetJobRequest
extends
CloudamRequest
<
GetJobResponse
>
{
private
final
String
urlSuffix
=
CloudamURL
.
GET_JOB_URL_SUFFIX
;
private
String
jobId
;
private
String
masterId
;
public
String
getJobId
()
{
return
jobId
;
}
public
void
setJobId
(
String
jobId
)
{
this
.
jobId
=
jobId
;
}
public
String
getMasterId
()
{
return
masterId
;
}
public
void
setMasterId
(
String
masterId
)
{
this
.
masterId
=
masterId
;
}
@Override
public
Class
<
GetJobResponse
>
getResponseClass
()
{
return
GetJobResponse
.
class
;
}
@Override
public
Request
.
Builder
getBuilder
(
Request
.
Builder
builder
,
String
url
,
CloudamRequest
<
GetJobResponse
>
cloudamRequest
)
{
GetJobRequest
getJobRequest
=
(
GetJobRequest
)
cloudamRequest
;
return
builder
.
url
(
url
+
getJobRequest
.
urlSuffix
+
"?requestId="
+
getJobRequest
.
getJobId
())
.
get
();
}
}
src/main/java/com/cloudam/cromwell/core/GetJobResponse.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
core
;
import
com.cloudam.cromwell.CloudamResponse
;
import
com.cloudam.cromwell.domain.Job
;
public
class
GetJobResponse
extends
CloudamResponse
{
}
src/main/java/com/cloudam/cromwell/core/WorkSpaceFactory.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
core
;
import
com.cloudam.cromwell.constant.CloudamURL
;
import
java.util.HashMap
;
import
java.util.Map
;
public
class
WorkSpaceFactory
{
private
final
static
Map
<
String
,
String
>
WORKSPACE_MAP
=
new
HashMap
<
String
,
String
>(){
{
put
(
"dev"
,
CloudamURL
.
BASE_DEV_URL
);
put
(
"release"
,
CloudamURL
.
BASE_RELEASE_URL
);
put
(
"prod"
,
CloudamURL
.
BASE_PROD_URL
);
}
};
public
static
String
getBaseUrl
(
String
env
){
return
WORKSPACE_MAP
.
get
(
env
);
}
}
src/main/java/com/cloudam/cromwell/domain/AutoCluster.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
domain
;
/**
* @author lglhope
*
* 启动集群的队列、实例类型、实例操作系统、磁盘
*/
public
class
AutoCluster
{
/**
* 队列(c-4-1)
*/
private
String
partition
;
/**
* 实例类型(ONDEMAND|SPOT)
*/
private
String
type
;
/**
* 实例操作系统(LINUX|WINDOWS)
*/
private
Os
os
=
Os
.
Linux
;
/**
* 实例磁盘类型及大小(SSD 40)
*/
private
CLoudamDisk
disk
;
public
String
getPartition
()
{
return
partition
;
}
public
void
setPartition
(
String
partition
)
{
this
.
partition
=
partition
;
}
public
String
getType
()
{
return
type
;
}
public
void
setType
(
String
type
)
{
this
.
type
=
type
;
}
public
Os
getOs
()
{
return
os
;
}
public
void
setOs
(
Os
os
)
{
this
.
os
=
os
;
}
}
\ No newline at end of file
src/main/java/com/cloudam/cromwell/domain/CLoudamDisk.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
domain
;
public
class
CLoudamDisk
{
private
String
type
;
private
int
size
;
public
CLoudamDisk
()
{
}
public
CLoudamDisk
(
String
type
,
int
size
)
{
this
.
type
=
type
;
this
.
size
=
size
;
}
}
src/main/java/com/cloudam/cromwell/domain/Command.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
domain
;
import
java.util.Map
;
public
class
Command
{
// autoCluster(queue(partition)、instanceCount(-N)、disk(tyepe and size)、os(LInux or Windows)、cpus(-c)、gpus(-G)、ntasks(-n)、username、zoneId),
// name, timeout, command(commandLine、envVars)
private
String
commandLine
;
private
Map
<
String
,
String
>
envVars
;
public
String
getCommandLine
()
{
return
commandLine
;
}
public
void
setCommandLine
(
String
commandLine
)
{
this
.
commandLine
=
commandLine
;
}
public
Map
<
String
,
String
>
getEnvVars
()
{
return
envVars
;
}
public
void
setEnvVars
(
Map
<
String
,
String
>
envVars
)
{
this
.
envVars
=
envVars
;
}
}
\ No newline at end of file
src/main/java/com/cloudam/cromwell/domain/Job.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
domain
;
public
class
Job
{
private
String
jobId
;
private
String
status
;
private
String
message
;
private
String
requestId
;
private
String
errorCode
;
private
String
jobStatus
;
public
Job
()
{
}
public
Job
(
String
jobId
,
String
status
,
String
message
,
String
requestId
,
String
errorCode
,
String
jobStatus
)
{
this
.
jobId
=
jobId
;
this
.
status
=
status
;
this
.
message
=
message
;
this
.
requestId
=
requestId
;
this
.
errorCode
=
errorCode
;
this
.
jobStatus
=
jobStatus
;
}
public
String
getJobId
()
{
return
jobId
;
}
public
void
setJobId
(
String
jobId
)
{
this
.
jobId
=
jobId
;
}
public
String
getStatus
()
{
return
status
;
}
public
void
setStatus
(
String
status
)
{
this
.
status
=
status
;
}
public
String
getMessage
()
{
return
message
;
}
public
void
setMessage
(
String
message
)
{
this
.
message
=
message
;
}
public
String
getRequestId
()
{
return
requestId
;
}
public
void
setRequestId
(
String
requestId
)
{
this
.
requestId
=
requestId
;
}
public
String
getErrorCode
()
{
return
errorCode
;
}
public
void
setErrorCode
(
String
errorCode
)
{
this
.
errorCode
=
errorCode
;
}
public
String
getJobStatus
()
{
return
jobStatus
;
}
public
void
setJobStatus
(
String
jobStatus
)
{
this
.
jobStatus
=
jobStatus
;
}
}
src/main/java/com/cloudam/cromwell/domain/JobDescription.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
domain
;
import
java.util.Map
;
/**
* @author lglhope
*
* 作业描述,用于配置生成sbatch.submit文件执行命令
*/
public
class
JobDescription
{
/**
* 作业名称
*/
private
String
name
;
/**
* 计算区ID
*/
private
String
zoneId
;
/**
* 计算区ID
*/
private
String
projectId
;
/**
* 集群配置
*/
private
AutoCluster
autoCluster
;
/**
* 环境变量
*/
private
Map
<
String
,
String
>
envs
;
/**
* CPUs核数(-c)
*/
private
int
cpus
;
/**
* GPUs核数(-g)
*/
private
int
gpus
;
/**
* 任务数(-n)
*/
private
int
ntasks
;
/**
* 超时时间(-t)
*/
private
int
timeout
;
public
String
getName
()
{
return
name
;
}
public
void
setName
(
String
name
)
{
this
.
name
=
name
;
}
public
String
getZoneId
()
{
return
zoneId
;
}
public
void
setZoneId
(
String
zoneId
)
{
this
.
zoneId
=
zoneId
;
}
public
String
getProjectId
()
{
return
projectId
;
}
public
void
setProjectId
(
String
projectId
)
{
this
.
projectId
=
projectId
;
}
public
AutoCluster
getAutoCluster
()
{
return
autoCluster
;
}
public
void
setAutoCluster
(
AutoCluster
autoCluster
)
{
this
.
autoCluster
=
autoCluster
;
}
public
Map
<
String
,
String
>
getEnvs
()
{
return
envs
;
}
public
void
setEnvs
(
Map
<
String
,
String
>
envs
)
{
this
.
envs
=
envs
;
}
public
int
getCpus
()
{
return
cpus
;
}
public
void
setCpus
(
int
cpus
)
{
this
.
cpus
=
cpus
;
}
public
int
getGpus
()
{
return
gpus
;
}
public
void
setGpus
(
int
gpus
)
{
this
.
gpus
=
gpus
;
}
public
int
getNtasks
()
{
return
ntasks
;
}
public
void
setNtasks
(
int
ntasks
)
{
this
.
ntasks
=
ntasks
;
}
public
int
getTimeout
()
{
return
timeout
;
}
public
void
setTimeout
(
int
timeout
)
{
this
.
timeout
=
timeout
;
}
}
src/main/java/com/cloudam/cromwell/domain/MountEntry.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
domain
;
public
class
MountEntry
{
public
String
getDestination
()
{
return
destination
;
}
public
void
setDestination
(
String
destination
)
{
this
.
destination
=
destination
;
}
public
String
getSource
()
{
return
Source
;
}
public
void
setSource
(
String
source
)
{
Source
=
source
;
}
public
boolean
isWriteSupport
()
{
return
writeSupport
;
}
public
void
setWriteSupport
(
boolean
writeSupport
)
{
this
.
writeSupport
=
writeSupport
;
}
private
String
destination
;
private
String
Source
;
private
boolean
writeSupport
;
}
\ No newline at end of file
src/main/java/com/cloudam/cromwell/domain/Os.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
domain
;
public
enum
Os
{
Linux
,
Windows
;
}
src/main/java/com/cloudam/cromwell/http/HttpUtil.java
0 → 100644
View file @
a4e75954
package
com
.
cloudam
.
cromwell
.
http
;
import
com.cloudam.cromwell.CloudamClient
;
import
okhttp3.Call
;
import
okhttp3.OkHttpClient
;
import
okhttp3.Request
;
import
okhttp3.Response
;
import
org.slf4j.Logger
;
import
org.slf4j.LoggerFactory
;
import
java.io.IOException
;
import
java.util.concurrent.TimeUnit
;
public
class
HttpUtil
{
private
final
static
Logger
logger
=
LoggerFactory
.
getLogger
(
HttpUtil
.
class
);
public
static
Request
.
Builder
getBuilder
(
String
accessToken
)
{
return
new
Request
.
Builder
()
.
addHeader
(
"Content-Type"
,
"application/json"
)
.
addHeader
(
"authorization"
,
"Bearer "
+
accessToken
);
}
public
static
Response
execNormalHttpRequest
(
Request
.
Builder
builder
,
int
MAX_RETRIES
)
{
Request
request
=
builder
.
build
();
int
retries
=
0
;
while
(
retries
<
MAX_RETRIES
)
{
try
{
OkHttpClient
okHttpClient
=
new
OkHttpClient
.
Builder
()
.
readTimeout
(
600
,
TimeUnit
.
SECONDS
)
.
writeTimeout
(
60
,
TimeUnit
.
SECONDS
)
.
connectTimeout
(
5
,
TimeUnit
.
SECONDS
).
build
();
final
Call
call
=
okHttpClient
.
newCall
(
request
);
Response
response
=
call
.
execute
();
if
(
response
==
null
||
response
.
code
()
!=
200
)
{
logger
.
error
(
"fail to execHttpRequest:{} ,url: {} "
,
"code not equal 200"
,
request
.
url
().
uri
().
toString
());
}
return
response
;
}
catch
(
IOException
e
)
{
logger
.
error
(
"fail to execHttpRequest ,reason: {} "
,
e
.
getMessage
(),
e
);
retries
++;
try
{
Thread
.
sleep
(
3000
);
}
catch
(
InterruptedException
ex
)
{
}
}
}
return
null
;
}
}
src/main/resources/application.properties
0 → 100644
View file @
a4e75954
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment